Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[ramicio]

Can anyone please help me with a new setup? I previously used a UDM-Pro and it did all of my routing. I am going with an OPNSense router. I have a bunch of VLANs on the switch, and I want the switch to do the routing. It could be a lot of traffic from any of those VLANs to my main one. I am testing VLAN 112, with an access point plugged into it. Connecting to it I am able to pull an IP address. I am unable to access the internet. I am unable to find any information on how to get.

Current configuration:
!
ver 08.0.30uT7f3
!
stack unit 1
  module 1 icx6610-24p-poe-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
  stack-trunk 1/2/6 to 1/2/7
  stack-port 1/2/6
stack unit 2
  module 1 icx6610-24f-sf-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
  stack-trunk 2/2/1 to 2/2/2
  stack-trunk 2/2/6 to 2/2/7
  stack-port 2/2/1 2/2/6
stack enable
stack mac 748e.f8fe.93f6
!
global-stp
!
!
!
vlan 1 name DEFAULT-VLAN by port
 router-interface ve 1
!
vlan 101 name Staab by port
 tagged ethe 1/3/8 ethe 2/1/1
!
vlan 102 name Jones by port
 tagged ethe 1/3/8 ethe 2/1/2
!
vlan 103 name Wagner by port
 tagged ethe 1/3/8 ethe 2/1/3
!
vlan 104 name Beamesderfer by port
 tagged ethe 1/3/8 ethe 2/1/4
!
vlan 105 name Sowers by port
 tagged ethe 1/3/8 ethe 2/1/5
!
vlan 106 name Speece by port
 tagged ethe 1/3/8 ethe 2/1/6
!
vlan 107 name SoLeb-Broseph by port
 tagged ethe 1/3/8 ethe 2/1/7
!
vlan 108 name DuPont-Harding by port
 tagged ethe 1/3/8 ethe 2/1/8
!
vlan 109 name Gromis by port
 tagged ethe 1/3/8 ethe 2/1/9
!
vlan 110 name Swanger by port
 tagged ethe 1/3/8 ethe 2/1/10
!
vlan 111 name Hackman by port
 tagged ethe 1/3/8 ethe 2/1/11
!
vlan 112 name Miranda by port
 tagged ethe 1/3/8 ethe 2/1/12
 router-interface ve 112
!
!
!
!
!
aaa authentication web-server default local
aaa authentication login default local
hostname switcheroo
ip dhcp-client disable
ip dhcp-server enable
!
ip dhcp-server pool miranda
 dhcp-default-router 192.168.112.1
 dns-server 192.168.112.1
 excluded-address 192.168.112.1
 lease 1 0 0
 network 192.168.112.0 255.255.255.0
 deploy
!
!
no telnet server
username root password .....
!
!
hitless-failover enable
!
!
!
!
!
!
!
interface ethernet 1/1/3
 inline power
!
interface ethernet 1/1/6
 inline power
!
interface ethernet 1/1/7
 inline power
!
interface ethernet 1/1/13
 inline power
!
interface ethernet 1/3/1
 port-name Indoors
 speed-duplex 10G-full
!
interface ethernet 1/3/2
 speed-duplex 10G-full
!
interface ethernet 1/3/3
 speed-duplex 10G-full
!
interface ethernet 1/3/4
 speed-duplex 10G-full
!
interface ethernet 1/3/7
 speed-duplex 10G-full
!
interface ethernet 1/3/8
 dual-mode
 speed-duplex 10G-full
!
interface ethernet 2/1/1
 port-name Staab
 dual-mode
!
interface ethernet 2/1/2
 port-name Jones
 dual-mode
!
interface ethernet 2/1/3
 port-name Wagner
 dual-mode
!
interface ethernet 2/1/4
 port-name Beamesderfer
 dual-mode
!
interface ethernet 2/1/5
 port-name Sowers
 dual-mode
!
interface ethernet 2/1/6
 port-name Speece
 dual-mode
!
interface ethernet 2/1/7
 port-name SoLeb-Berger
 dual-mode
!
interface ethernet 2/1/8
 port-name DuPont-Harding
 dual-mode
!
interface ethernet 2/1/9
 port-name Gromis
 dual-mode
!
interface ethernet 2/1/10
 port-name Swanger
 dual-mode
!
interface ethernet 2/1/11
 port-name Hackman
 dual-mode
!
interface ethernet 2/1/12
 port-name Miranda
 dual-mode
!
interface ve 1
 ip address 192.168.1.5 255.255.255.0
!
interface ve 112
 ip address 192.168.112.1 255.255.255.0
!
!
!
!
!
lldp tagged-packets process
lldp run
!
!
!
!
end

 

[cyinite]

Can anyone please help me with a new setup? I previously used a UDM-Pro and it did all of my routing. I am going with an OPNSense router. I have a bunch of VLANs on the switch, and I want the switch to do the routing. It could be a lot of traffic from any of those VLANs to my main one. I am testing VLAN 112, with an access point plugged into it. Connecting to it I am able to pull an IP address. I am unable to access the internet. I am unable to find any information on how to get.

Current configuration:
!
ver 08.0.30uT7f3
!
stack unit 1
  module 1 icx6610-24p-poe-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
  stack-trunk 1/2/6 to 1/2/7
  stack-port 1/2/6
stack unit 2
  module 1 icx6610-24f-sf-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
  stack-trunk 2/2/1 to 2/2/2
  stack-trunk 2/2/6 to 2/2/7
  stack-port 2/2/1 2/2/6
stack enable
stack mac 748e.f8fe.93f6
!
global-stp
!
!
!
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
!
vlan 101 name Staab by port
tagged ethe 1/3/8 ethe 2/1/1
!
vlan 102 name Jones by port
tagged ethe 1/3/8 ethe 2/1/2
!
vlan 103 name Wagner by port
tagged ethe 1/3/8 ethe 2/1/3
!
vlan 104 name Beamesderfer by port
tagged ethe 1/3/8 ethe 2/1/4
!
vlan 105 name Sowers by port
tagged ethe 1/3/8 ethe 2/1/5
!
vlan 106 name Speece by port
tagged ethe 1/3/8 ethe 2/1/6
!
vlan 107 name SoLeb-Broseph by port
tagged ethe 1/3/8 ethe 2/1/7
!
vlan 108 name DuPont-Harding by port
tagged ethe 1/3/8 ethe 2/1/8
!
vlan 109 name Gromis by port
tagged ethe 1/3/8 ethe 2/1/9
!
vlan 110 name Swanger by port
tagged ethe 1/3/8 ethe 2/1/10
!
vlan 111 name Hackman by port
tagged ethe 1/3/8 ethe 2/1/11
!
vlan 112 name Miranda by port
tagged ethe 1/3/8 ethe 2/1/12
router-interface ve 112
!
!
!
!
!
aaa authentication web-server default local
aaa authentication login default local
hostname switcheroo
ip dhcp-client disable
ip dhcp-server enable
!
ip dhcp-server pool miranda
dhcp-default-router 192.168.112.1
dns-server 192.168.112.1
excluded-address 192.168.112.1
lease 1 0 0
network 192.168.112.0 255.255.255.0
deploy
!
!
no telnet server
username root password .....
!
!
hitless-failover enable
!
!
!
!
!
!
!
interface ethernet 1/1/3
inline power
!
interface ethernet 1/1/6
inline power
!
interface ethernet 1/1/7
inline power
!
interface ethernet 1/1/13
inline power
!
interface ethernet 1/3/1
port-name Indoors
speed-duplex 10G-full
!
interface ethernet 1/3/2
speed-duplex 10G-full
!
interface ethernet 1/3/3
speed-duplex 10G-full
!
interface ethernet 1/3/4
speed-duplex 10G-full
!
interface ethernet 1/3/7
speed-duplex 10G-full
!
interface ethernet 1/3/8
dual-mode
speed-duplex 10G-full
!
interface ethernet 2/1/1
port-name Staab
dual-mode
!
interface ethernet 2/1/2
port-name Jones
dual-mode
!
interface ethernet 2/1/3
port-name Wagner
dual-mode
!
interface ethernet 2/1/4
port-name Beamesderfer
dual-mode
!
interface ethernet 2/1/5
port-name Sowers
dual-mode
!
interface ethernet 2/1/6
port-name Speece
dual-mode
!
interface ethernet 2/1/7
port-name SoLeb-Berger
dual-mode
!
interface ethernet 2/1/8
port-name DuPont-Harding
dual-mode
!
interface ethernet 2/1/9
port-name Gromis
dual-mode
!
interface ethernet 2/1/10
port-name Swanger
dual-mode
!
interface ethernet 2/1/11
port-name Hackman
dual-mode
!
interface ethernet 2/1/12
port-name Miranda
dual-mode
!
interface ve 1
ip address 192.168.1.5 255.255.255.0
!
interface ve 112
ip address 192.168.112.1 255.255.255.0
!
!
!
!
!
lldp tagged-packets process
lldp run
!
!
!
!
end

Welcome from Ubiquiti, their networking equipment definitely is as easy as it gets but congrats for making the leap. Looking around your running config, I noticed you might be missing everything you need. From my experience, I needed to run

1. ip route 0.0.0.0/0 [opnsense ip] on the switch
2. add the switches IP as a gateway in OPNsense under System > Gateways > Configuration
3. add the VLAN subnets to route through the switch under System > Routes > Configuration
4. finally add a outbound rule under Firewall > NAT > Outbound so that OPNsense NATs the VLANs

You are right about this being hard to find info for, I was only able to do this once I learned about it studying for my CCNA.

 

[ramicio]

Welcome from Ubiquiti, their networking equipment definitely is as easy as it gets but congrats for making the leap. Looking around your running config, I noticed you might be missing everything you need. From my experience, I needed to run

1. ip route 0.0.0.0/0 [opnsense ip] on the switch
2. add the switches IP as a gateway in OPNsense under System > Gateways > Configuration
3. add the VLAN subnets to route through the switch under System > Routes > Configuration
4. finally add a outbound rule under Firewall > NAT > Outbound so that OPNsense NATs the VLANs

You are right about this being hard to find info for, I was only able to do this once I learned about it studying for my CCNA.

Thank you. I will have to test it at a bit later point in time. Currently I have some pieces of hardware in use for keeping my UDM Pro connected to my modem because I am still using the Protect portion of it (pieces I was using before to test this Brocade thing). Until I get a UNVR here... That will be the only thing left here from Ubiquiti.

I'm not really understanding what any of that means after item #1.

Thank you.

 

[cyinite]

Thank you. I will have to test it at a bit later point in time. Currently I have some pieces of hardware in use for keeping my UDM Pro connected to my modem because I am still using the Protect portion of it (pieces I was using before to test this Brocade thing). Until I get a UNVR here... That will be the only thing left here from Ubiquiti.

I'm not really understanding what any of that means after item #1.

Thank you.

No worries. Looking back when I first started learning, fundamentals like routing were pretty intimating. Here's some resources to help with configuration:

• Gateways — OPNsense documentation
• Routes — OPNsense documentation
• Network Address Translation — OPNsense documentation

Best of luck to you!

 

[ramicio]

No worries. Looking back when I first started learning, fundamentals like routing were pretty intimating. Here's some resources to help with configuration:

• Gateways — OPNsense documentation
• Routes — OPNsense documentation
• Network Address Translation — OPNsense documentation

Best of luck to you!

I was watching those videos yesterday and the first VLAN one seems like what I already had been doing. The OPNSense stuff still makes zero sense to me. I am the type of person who needs to find someone who is doing the literal same thing as I am, and copying/pasting and inserting my own addresses. I am doing this for a few reasons. First, Ubiquiti and their insane security stuff...I don't need to reach this stuff outside of my network. So the whole using my UI account to get into my own local stuff, and needing to do 2FA for it is insane, along with it all needing to be encrypted. I can do plain HTTP with all of this new stuff so far, Omada being what I am now using for access points. My goal was to offload routing to the switch in case of local Plex traffic. I'd rather it not have to travel to the internet router and back into the switch, when the switch could do it. Literally the only traffic. My internet is 2.5 gigabits, so that leaves 7.5 gigabits for all that other traffic, so in reality, it would never get congested. With the Ubiquiti stuff, people were still seeing other's TVs broadcasting for being able to cast videos to them. This is at a campground where I live, where a few people get internet access from me, versus the sparse access points outdoors that the camp itself has set up. I'd preferably just pay someone to set this up for me, but have never gotten anyone interested in some side cash (I don't get it, seems like an easy gig, not setting up a whole company's infrastructure).

 

[kpfleming]

I'd preferably just pay someone to set this up for me, but have never gotten anyone interested in some side cash (I don't get it, seems like an easy gig, not setting up a whole company's infrastructure).

Speaking personally, I avoid doing this because it's not a one-time interaction; once it has been setup, any time there's an issue, or there's a need for a security patch to be applied, or something wonky happens with the upstream link(s), the person who did the 'side gig' is going to get a call.

 

[ramicio]

Speaking personally, I avoid doing this because it's not a one-time interaction; once it has been setup, any time there's an issue, or there's a need for a security patch to be applied, or something wonky happens with the upstream link(s), the person who did the 'side gig' is going to get a call.

That makes zero sense to me. Setting it up properly either works or it doesn't work. The person who did the side gig would be getting paid again if I needed more help anyway. Regardless, I guess I am going to give up. And I am probably going to just go back to the UDM Pro. I restarted this OPNSense box and one of the interfaces of the dual-port NIC just totally disappeared from existence. That, and I can't get any help with setting this up.

 

[BoGs]

If you are willing to give up that easily unfortunately this was not going to work out, there is no nice UI that you can use for these older switches. You will need to change some settings unplug the SFP plugs from your other switch and test it out. Unless you work in that field which most of us do not. It took me 2 months to switch from OPNSense to Mikrotik router as I wanted to route more then 2.5G and instead of building a new server I went for my largest link SFP+.

The reason why people are saying not to do side gig is so that you can learn, and ultimately success and being proud of what you achieve.

What you want to do out of the gate is more advanced then most, what you are looking for is to setup a transit VLAN between your switch and router, your switch becomes the router by setting up routable interfaces and you create a route on the switch for other vlan subnets to pass it to OPNSense and on the OPNSense to pass it to the switch. You will also have to pass the DHCP relay to wherever your server is.

I used this to read up on intervlan routing (InterVLAN Routing) or this (Inter-VLAN Routing Configuration - %%currentyear%% Step-by-Step Tutorial) and while I think its for Cisco you can lookup the keyword commands in the brocade command reference sheet. Keeping router on a stick is probably the easiest option as you can setup your OPNSense vlans on top of the interface and away you go with trunk ports. I would do that FIRST then potentially experimenting with _new_ vlans on different ports so you do not break the internet.

Good luck

 

[ramicio]

If you are willing to give up that easily unfortunately this was not going to work out, there is no nice UI that you can use for these older switches. You will need to change some settings unplug the SFP plugs from your other switch and test it out. Unless you work in that field which most of us do not. It took me 2 months to switch from OPNSense to Mikrotik router as I wanted to route more then 2.5G and instead of building a new server I went for my largest link SFP+.

The reason why people are saying not to do side gig is so that you can learn, and ultimately success and being proud of what you achieve.

What you want to do out of the gate is more advanced then most, what you are looking for is to setup a transit VLAN between your switch and router, your switch becomes the router by setting up routable interfaces and you create a route on the switch for other vlan subnets to pass it to OPNSense and on the OPNSense to pass it to the switch. You will also have to pass the DHCP relay to wherever your server is.

I used this to read up on intervlan routing (InterVLAN Routing) or this (Inter-VLAN Routing Configuration - %%currentyear%% Step-by-Step Tutorial) and while I think its for Cisco you can lookup the keyword commands in the brocade command reference sheet. Keeping router on a stick is probably the easiest option as you can setup your OPNSense vlans on top of the interface and away you go with trunk ports. I would do that FIRST then potentially experimenting with _new_ vlans on different ports so you do not break the internet.

Good luck

I cannot learn how to do this unless someone literally shows me how to do this. Documents and videos are of no help to me. I don't understand the stuff. I only understand by seeing how it is done. I cannot experiment with more ports. My OPNSense router is 1u and has only a dual-port SFP+ NIC. Impossible.

 

[DavidVermillion]

I cannot learn how to do this unless someone literally shows me how to do this. Documents and videos are of no help to me. I don't understand the stuff. I only understand by seeing how it is done. I cannot experiment with more ports. My OPNSense router is 1u and has only a dual-port SFP+ NIC. Impossible.

That's why no one wants to configure your stuff as a "side gig." A) The knowledge is fairly esoteric and B) people who refuse to learn and just want someone else to do it all for them or feed them step by step instructions usually end being combative, demanding, and want far more support.

Sorry, my dude, but nothing in your tone sounds like you'd be an exception to this. My spider-sense from back when I did freelance IT is tingling. Good luck, I hope you can find someone to do it for you.

 

[ramicio]

That's why no one wants to configure your stuff as a "side gig." A) The knowledge is fairly esoteric and B) people who refuse to learn and just want someone else to do it all for them or feed them step by step instructions usually end being combative, demanding, and want far more support.

Sorry, my dude, but nothing in your tone sounds like you'd be an exception to this. My spider-sense from back when I did freelance IT is tingling. Good luck, I hope you can find someone to do it for you.

Such a weird attitude, makes no logical sense. People are fine punching a clock to do that very same IT work, and no one there wants to know how the sausage is made. Meanwhile, someone who want the same help, want to know what they did, with a quick explainer, nope. Taboo. And it's not even esoteric. It's basic IT stuff. 1. What's combative? 2. What's demanding? 3. Where did I say I want support (and you're sounding like you're mocking me as if I'd want it for free, when I've said the opposite)? Not being combative and demanding. I don't understand how to do this, and posting generic articles for an idiot like me to read isn't helpful. The same effort spent to do that could just have been spent to show me literally how to do it. It's just industrial gatekeeping as far as I'm concerned.

 

[bwahaha]

@ramicio you may not see the behavior they were referring to, but we can.
You didn't ask for a quick explanation, you asked for a hand holding.

1) what combative? well, that post, as example....
2) What's demanding? well that post, as example...
3) whats.... that? well, that post for example.

We get it, something seemingly simple, should be a simple answer. Unfortunately, you were given solid information and you said rejected it.

We aren't on a Professional Support forum, we're a hobbyist learning forum. Many of us are, were, or will be professionals, so when "outside of scope of homelab" pops in, and demands professional support, for free (yes, you offered to pay), and becomes combative, it irks us. We've dealt with demanding customers, clients, users, and we come here to get away from them.

 

[Cheburashka]

Why do you want to run jumbo frames in 2024? They're almost entirely unnecessary for 10gb, have you done any throughput testing etc showing that you need them for some reason?

I guess I was trying to figure out why between my QNAP, the ICX SW and my W11 desktop, am I getting 5.6gig with iperf. I was just going through threads and trying to identify ways to improve the performance of the transfer.

I have to do this test from a linux live boot ISO instead of Windows to see if the issue is within the OS or not.

 

[hmw]

I guess I was trying to figure out why between my QNAP, the ICX SW and my W11 desktop, am I getting 5.6gig with iperf. I was just going through threads and trying to identify ways to improve the performance of the transfer.

I have to do this test from a linux live boot ISO instead of Windows to see if the issue is within the OS or not.

jumbo frames might help by 10% but unless you're tunneling or encapsulating other protocols, they're almost never the issue.

try running iperf in parallel mode and see if you can saturate 10g by running 2-3 iperf streams in parallel ( for example iperf3 -P 3 ). Windows is notorious for not being able to saturate 10G with just one connection/thread. The other problem is something in between that might cause impedance mismatch like a SFP to 10GBase-T transceiver etc

 

[86turbodsl]

can anyone help me understand what a proper stack setup on a 6610 is supposed to look like? I have a 6610 48p in the house, and a 6610 24p in the shop. I ran stack secure-setup, it has a connection, it's complaining about a link. I thought one of the fibers might be mixed up because i've swapped sides a few times trying to get the link up. It's currently showing

One port, the left as green on each end. I am using 1/2/6 and 1/2/7 (bottom ports) on the big one and 3/2/1 and 3/2/2 (bottom) on the other end. 1/2/6 and 3/2/1 are showing as green currently. Should i assume i have a dead fiber if i tried switching sides on the one end of one of the fibers. Or swap optics out?

Stack Details

Unit ID	Type	Role	Mac Address	Priority	State	Comment
1	S ICX6610-48P	active	748e.f8fb.f0a2	128	local	Ready
3	D ICX6610-24P	standby	cc4e.24c5.b6fc	0	remote	Ready

alone: standalone, D: dynamic config, S: static config


Stack Port Status

Unit ID	Stack-port1	Stack-port2
1	dn (1/2/1)	up (1/2/6)
3	up (3/2/1-3/2/2)	dn (3/2/6-3/2/7)

Stack Neighbors

Unit ID	Stack-port1	Stack-port2
1	none	3
3	1	none

 

[DouglasteR]

Hi there guys,

The time has come and i´m on the verge to buy the 24p POE+ version and setup it as my main switch/dhcp in my home.

But i have one question first, can i connect my ConnectX3 - QSFP+ directly to the switch and connect at 40Gbps ?

My use case is learning switching, power some ip cameras and common soho networking.

Thanks.

 

[BBergle]

Sometimes the issue is bad serial cable. Some of the Arista and Celestica switches I bought came with their own serial cables, and they work much better than old one I had.

I am actually using a serial cable from an Arista switch I got on ebay. The seller sent me a new switch and I am having the same issue. It has to be one of the cables/adapters I am using. I feel like im going insane. I have tried three different computers. M2 Macbook pro, Intel Macbook Pro and a windows PC and they all have the same issue but the windows computer has horizontal lines instead of question marks. I have new cables coming tomorrow but my current ones work fine for my Arista switch so I really don't know what's going on. All I can seem to get on my terminal screen are these characters ����

 

[kpfleming]

Hi there guys,

The time has come and i´m on the verge to buy the 24p POE+ version and setup it as my main switch/dhcp in my home.

But i have one question first, can i connect my ConnectX3 - QSFP+ directly to the switch and connect at 40Gbps ?

My use case is learning switching, power some ip cameras and common soho networking.

Thanks.

If by "the 24p POE+" version you mean the 7250-24P, then no, the high speed ports are 10Gbps maximum.

 

[DouglasteR]

I was talking about the ICX6610-24p !

My idea was using the back QSFP ports as 40G uplink !

 

[fohdeesha]

I was talking about the ICX6610-24p !

My idea was using the back QSFP ports as 40G uplink !

two of them yes as my docu explains. the other two are 43x 10gb breakout. also you do not want to use the built in DHCP server on these things