Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

aaroneaton

New Member
Jan 15, 2021
12
0
1
www.rfehosting.com
After a whole lot of tests i was able to find out that the ports are bad on the switch itself.. Fun times!


please post your config. Yeah, I know you said you have stacking disabled - but worth a look.

For the rear connections:

I'm currently using AOC QSFP to SFP+ breakouts (generic/cisco) and they work great. I've tested generic QSFP to SFP+ DAC - worked fine too.
I've used netapp QSFP to QSFP for stacking on both the 40gbe and breakouts. - worked fine
I've used AOC QSFP to QSFP for switch to host - that also worked fine.

Nothing magical in the cables I've used - in fact they were the cheapest I could find of their respective types - configs were pretty much stock on the 1/2/x side.

when you say reg fs DAC cables work, going from the front sfp+ connections to the servers?

what did show int give you on the breakout ports when you were testing?

for example here's a snippet from my show run on a standalone icx6610 and as you can see nothing configured for my 1/2/x ports and they all work for hosts:

Code:
!
lag LAG41 dynamic id 41
ports ethernet 1/2/1 ethernet 1/2/6
primary-port 1/2/6
deploy
!
...
!
interface ethernet 1/1/24
dual-mode  249
inline power
!
interface ethernet 1/3/1
speed-duplex 10G-full
!
2 10gbe in use, 40gbe configured but host is down at the moment. That's with a generic 40gbe to 10gbe AOC breakout.

Code:
SSH@icx6610-stack#show inter br ethe 1/2/1 to 1/2/10

Port       Link    State   Dupl Speed Trunk Tag Pvid Pri MAC             Name
1/2/1      Down    None    None None  41    Yes N/A  0   748e.f8dc.ae80                
1/2/2      Down    None    None None  None  Yes N/A  0   748e.f8dc.ae80                
1/2/3      Down    None    None None  None  Yes N/A  0   748e.f8dc.ae80                
1/2/4      Down    None    None None  None  Yes N/A  0   748e.f8dc.ae80                
1/2/5      Up      Forward Full 10G   None  Yes N/A  0   748e.f8dc.ae80                
1/2/6      Down    None    None None  41    Yes N/A  0   748e.f8dc.ae80                
1/2/7      Down    None    None None  None  Yes N/A  0   748e.f8dc.ae80                
1/2/8      Down    None    None None  None  Yes N/A  0   748e.f8dc.ae80                
1/2/9      Down    None    None None  None  Yes N/A  0   748e.f8dc.ae80                
1/2/10     Up      Forward Full 10G   None  Yes N/A  0   748e.f8dc.ae80                
SSH@icx6610-stack#
 
  • Wow
Reactions: itronin

LodeRunner

Active Member
Apr 27, 2019
540
227
43
Has anyone seen this boot message before?
Code:
soc_do_init: soc_do_init: total cpu and arm cosq 0030 unexpected
Getting that on a 7450 after updating to 8090UFI.
 

itronin

Well-Known Member
Nov 24, 2018
1,234
793
113
Denver, Colorado
dang! what is another switch that has the dual 40gb Qsfp with breakout comparability? that may be easier to get?
o_O Another ICX6610

Not sure if you are in the US but there are a ton of listings on the bay - you might also post in the WTB thread and someone may have one for you too? think of the one you have as a parts donor for fan(s) and PSU(S)
 

noduck

Member
Sep 12, 2020
38
10
8
It looks like ERSPAN does not seem to find new outgoing port after STP change.

I have an ERSPAN session configured from a 7150-c12. Normally, this traffic flows through an upstream 7250 switch. However, if that switch is not available there is another path that becomes available through STP.

ping/sFlow all work in both situations. ERSPAN can work in either situation, but will not flip itself (need to reboot switch, or recreate the ERSPAN configuration).

Is this an ICX bug? If so, anyway to report it?

Code:
#show erspan 
 
 Profile 3 
 Type             ERSPAN 
 Mirror destination Reachable. 
 Destination IP   172.31.x.y
 Destination MAC  52xx.xxxx.xxxx
 Source IP        172.31.x.z
 Source MAC       78xx.xxxx.xxxx
 Outgoing port    1/3/1 
 Outgoing VLAN    999 
 Outgoing VE      999 

#show arp | include 172.31.x.y
2     172.31.x.y       52xx.xxxx.xxxx Dynamic  2    1/1/9             Valid 
#show version 
  Copyright (c) Ruckus Networks, Inc. All rights reserved. 
    UNIT 1: compiled on Dec 28 2020 at 20:20:54 labeled as SPR08092e 
      (33554432 bytes) from Primary SPR08092e.bin (UFI) 
        SW: Version 08.0.92eT213
 

ZFSZealot

New Member
Aug 16, 2021
26
6
3
o_O Another ICX6610

Not sure if you are in the US but there are a ton of listings on the bay - you might also post in the WTB thread and someone may have one for you too? think of the one you have as a parts donor for fan(s) and PSU(S)
This. Those fan trays and PSU's bring an absurd amount of money on eBay if you can find them.
 

nw60312

New Member
Jul 28, 2021
6
0
1
Setting up a new ICX 7150-24p. Out of the box 1/2/1 is up can SSH into the box, upgrade firmware etc. Cannot get the USB console to work, or OOB network port. Would like to create a LAG on both uplink ports to the router. Can create the lag and add 1/2/2, however when I try to add 1/2/1 it fails as that has an IP address assigned (How I am able to SSH to the box). How do I get out of the race condition?

Anyone had success getting the USB console to work with a 2018 MacBook Pro? Tried installing the Ruckus drivers but those are blocked during install, tried version 6.0.1 direct for SI Labs but still no joy.
 

nickf1227

Active Member
Sep 23, 2015
197
128
43
33
I have only ever used a USB to DB9 serial adapter with a Cisco DB9 to RJ45 or Brocade DB9 to "mini-usb" for certain models on my MacBook. I used the program Serial. I would recommend investing in a console cable.
I've never tried any other method.

OOB interface maybe set to a static IP and not DHCP so that may be why you can't get in that way.

As for your other issue, the ICX line won't let you add two interfaces to a lag that have differing configurations. The easiest way to solve the problem you are having is to make a transit vlan.

Make up anew VLAN that will be used exclusively for routing between your router and your switch at L3. Tag that VLAN on the LAG between your switch and router, and on the LAG on your routers side. Make a VE on your Brocade in that VLAN and give it an address. Pick an address in that same subnet on the router side and assign it to the VLAN. You want to use a /30 for this type of link.

Setup your routing on both sides, and viola. You have separated your access network from your wan edge at L3. You can make this type of link as wide as you want. However, by the nature of doing things this way You are doing all of your inter-vlan routing in your Brocade switch and not sending that traffic north to your router for now reason.
 
Last edited:

mfolnovic

New Member
Jun 7, 2021
6
2
3
Hello everyone! I'm setting up ICX 6450-48P and I'm trying to follow @phil9878's instructions.

I'm beginner at this, but read as much as I could (first ~20 and last ~40 pages here and watched some of Terry Henry's videos), and I'm aware there are easier ways to use this switch (e.g. router on a stick, running isc-dhcp separately). I already have router on a stick running so I'm trying to learn and improve from there. I've also tried running isc-dhcp on separate VM (ip helper-address) and got basic setup working. I would have stayed at this setup if I didn't stumble upon @phil9878's instructions. :)

Current issue is my PCs (Windows and Linux) can't get IP address from pfsense's DHCP when I connect them to port 1/1/39. Pfsense is connected to port 1/1/1.

Cleaned up sh run:
Code:
vlan 1 name DEFAULT-VLAN by port
 router-interface ve 1
!
vlan 90 name Guest by port
 untagged ethe 1/1/39
 router-interface ve 90
!
vlan 300 by port
 tagged ethe 1/1/1
 router-interface ve 300
!
ip dhcp-client disable
ip route 0.0.0.0/0 172.26.1.1
!
interface ve 1
 ip address 192.168.2.1 255.255.255.0
!
interface ve 90
 ip address 192.168.90.2 255.255.255.0
!
interface ve 300
 ip address 172.26.1.2 255.255.255.0
And here are screenshots from pfsense:

1630240692007.png1630240737959.png1630240808725.png1630240836830.png

And DHCP server for VLAN 90 (didn't touch other options):
1630240872020.png1630240890874.png

VLAN 300 doesn't have DHCP server.

I've tried adding IP helper like:

Code:
interface ve 90
 ip address 192.168.90.2 255.255.255.0
 ip helper-address 1 192.168.90.1
but it didn't help.

Not sure how to debug this any further.
Thank you all very much for all the information on this forum and for future help!
 

itronin

Well-Known Member
Nov 24, 2018
1,234
793
113
Denver, Colorado
Hello everyone! I'm setting up ICX 6450-48P and I'm trying to follow @phil9878's instructions.

I'm beginner at this, but read as much as I could (first ~20 and last ~40 pages here and watched some of Terry Henry's videos), and I'm aware there are easier ways to use this switch (e.g. router on a stick, running isc-dhcp separately). I already have router on a stick running so I'm trying to learn and improve from there. I've also tried running isc-dhcp on separate VM (ip helper-address) and got basic setup working. I would have stayed at this setup if I didn't stumble upon @phil9878's instructions. :)

Current issue is my PCs (Windows and Linux) can't get IP address from pfsense's DHCP when I connect them to port 1/1/39. Pfsense is connected to port 1/1/1.
This is key. pfSense does NOT serve up DHCP except on directly connected interfaces.
I can't tell from your screenshots if you have defined your pfSense interface as VLAN tagged 90.
For sure you are not passing VLAN 90 traffic to PFsense at the switch since your configuration shows the only member is untagged ethe 1/1/39


!
vlan 90 name Guest by port
untagged ethe 1/1/39
router-interface ve 90
[/CODE]
you'll need to add this to your VLAN 90 definition and you'll need to make sure you have a VLAN 90 tagged (sub) interface on pfSense. this note is on the drawing in the instructions you linked above.

Code:
tagged ethe 1/1/1
I've tried adding IP helper like:

Code:
interface ve 90
ip address 192.168.90.2 255.255.255.0
ip helper-address 1 192.168.90.1
but it didn't help.

Not sure how to debug this any further.
Thank you all very much for all the information on this forum and for future help!
IP helper isn't going to do much for you because pfSense serves up DHCP on directly connected interfaces only.

My advice - break up your troubleshooting into functional components.

First test basic IP connectivity (this is more switch and pfSense interface config)
statically configure your PC with a an IP - say 192.168.90.9 (which is within your reserved DHCP block)
Based on your config you should be able to ping 192.168.90.2 since that is the switch's ve and ethe 1/1/39 is untagged in VLAN 90.
Once you can ping 192.168.90.2 move on to pinging 192.168.90.1 - as I said I can't see in your pfSense screenshots whether your VLAN 90 interface is actually defined correctly.

Second you can then move on to testing DHCP.
Then test routing directly out of pfSense first, make sure you have Internet access, NAT working correctly etc. Ie. define the def gw as the pfense box in your DHCP configuration.
Once that works move on to the heart of what you trying to do: which is use the ICX as the core router for LAN and and outbound Internet traffic via the transit VLAN ie. change the DHCP def gw for VLAN 90 pool to your ICX ve90 IP 192.168.90.2[/QUOTE]
 
  • Like
Reactions: mfolnovic

mfolnovic

New Member
Jun 7, 2021
6
2
3
Thank you very much for your help, setting missing tagged ethe 1/1/1 helped with pretty much everything!

DHCP now works:
1630248759119.png

So, everything (pinging public IP, pinging 192.168.90.1, reaching pfsense console) but DNS works, which is expected because I haven't even played with that yet.

With static IP, I couldn't ping 192.168.90.1 nor public IP. When I picked static IP from DHCP range, it worked.

change the DHCP def gw for VLAN 90 pool to your ICX ve90 IP 192.168.90.2
I think that's on last screenshot, right? Or did I set it up wrong?

Again, thank you very much! :)
 

Attachments

  • Like
Reactions: itronin

nw60312

New Member
Jul 28, 2021
6
0
1
Got OOB network running. What concepts am I missing to read up on. Device is running Fastiron 0900 layer 3 image. Uplink ports are configured into a LAG, only 1 cable connected at the moment. Both sides of the LAG show active. I can't get traffic to the switch uplink lag. I think this is the relevant config:



ver 09.0.00T213
!
stack unit 1
module 1 icx7150-24p-poe-port-management-module
module 2 icx7150-2-copper-port-2g-module
module 3 icx7150-4-sfp-plus-port-40g-module
!
global-stp
!
lag core dynamic id 1
ports ethe 1/2/1 to 1/2/2
disable ethe 1/2/2
!
vlan 1 name DEFAULT-VLAN by port
spanning-tree
!
chassis fanless 1
hostname core-switch
!

interface management 1
ip address 192.168.1.168 255.255.255.0
!

interface lag 1
ip address 172.16.10.5 255.255.255.0
!
!

no telnet server

!
end


 

nickf1227

Active Member
Sep 23, 2015
197
128
43
33
Can you type
show Ip Route?

Also wht is the config of the other side? What kind of device is it?
 

nw60312

New Member
Jul 28, 2021
6
0
1
Other end is an opnsense box. I think you hit the issue there is no route:

Code:
SSH@core-switch>show ip route


Total number of IP routes: 1


Type Codes - B:BGP D:Connected O:OSPF R:RIP S:Static; Cost - Dist/Metric


BGP  Codes - i:iBGP e:eBGP


OSPF Codes - i:Inter Area 1:External Type 1 2:External Type 2


STATIC Codes - v:Inter-VRF


        Destination        Gateway         Port          Cost          Type Uptime


1       192.168.1.0/24     DIRECT          e mgmt1       0/0           D    0m23s 


SSH@core-switch>
 

clcorbin

Member
Feb 15, 2014
38
7
8
This might be a silly question, but I haven't found an answer so far: Is there a way to permanently disable the paging, like the "skip-page-display" command does? For some reason that irritates me every single time I log in. Small problem, but if anyone has a solution...
 

nickf1227

Active Member
Sep 23, 2015
197
128
43
33
Other end is an opnsense box. I think you hit the issue there is no route:

Code:
SSH@core-switch>show ip route


Total number of IP routes: 1


Type Codes - B:BGP D:Connected O:OSPF R:RIP S:Static; Cost - Dist/Metric


BGP  Codes - i:iBGP e:eBGP


OSPF Codes - i:Inter Area 1:External Type 1 2:External Type 2


STATIC Codes - v:Inter-VRF


        Destination        Gateway         Port          Cost          Type Uptime


1       192.168.1.0/24     DIRECT          e mgmt1       0/0           D    0m23s


SSH@core-switch>
So you are not doing any routing on your switch?
Are you trying to connect at L2 or L3?

What does OPN Sense configuration look like?

What are you trying to do?