Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[seatrope]

Looking for some help from people here (esp @kapone you've been amazing in the past!).

I've gotten the standard Opnsense and ICX6610 setup working where all my local traffic between my 3 VLANs are routed internally, anything going to the internet is routed to OpnSense.

However, I just got a Reolink doorbell and very annoyingly, it will not allow local connections unless they appear to come from the same VLAN. The Reolink doorbell is on the IOT VLAN, and client devices connecting to the doorbell are usually on the "STANDARD" or "ADMIN" VLANs.

On the Reolink forums, the solution given was to Source NAT the client IP so that it appears to be coming from the same VLAN as the doorbell.

ICX switches cannot SNAT. So conceptually i think i have to route any traffic from the STANDARD or ADMIN VLANs directed to the Reolink Doorbell IP out to Opnsense for Opnsense to SNAT and send back into the IOT VLAN.

Before I start messing around blindly, I'm guessing i'd have to add an entry to the ICX6610 routing table and also some routing rules/SNAT rules in Opnsense.
Not sure of the exact details.

If someone could guide me on routing table modifications in the ICX6610 to achieve the above that would be great!

Thanks!!

 

[seatrope]

Well, i seem to lack a basic understanding of routing tables.

I added a static route for anything destined for the Reolink IP to be sent to Opnsense. Of course that sent anything destined for the Reolink back to Opnsense in an infinite loop, duh.

Is it possible to add a source - destination to the ICX routing tables? or is it just destination only?

 

[kapone]

I added a static route for anything destined for the Reolink IP to be sent to Opnsense

Why?

You have an L3 switch... VRFs are your friend.

 

[seatrope]

Why?

You have an L3 switch... VRFs are your friend.

lol. No idea where to begin with VRFs…

 

[kapone]

lol. No idea where to begin with VRFs…

Well...a few things we can try (without getting into VRFs...)

what "devices" on the STANDARD or ADMIN VLANs are we talking about? Are they VLAN capable natively? (It'd be trivial to do a dual homed device...IF they supported VLANs)

 

[seatrope]

Well...a few things we can try (without getting into VRFs...)

what "devices" on the STANDARD or ADMIN VLANs are we talking about? Are they VLAN capable natively? (It'd be trivial to do a dual homed device...IF they supported VLANs)

its ok I started reading about VRFs. New thing to learn!

the devices on the other VLANs are iPhones. Can’t dual home

 

[cyinite]

Hmm, I guess I'm confused then. It's been a while, but I don't recall having to do any routing to pass VLAN's between switches. I used to have some ancient Cisco switches and I seem to think that I could have say VLAN10 on both switches and assign ports on both. Do switch trunk allowed vlan all on the connection between them and devices connected to ports associated to the same VLAN at both ends could communicate with each other.

Sounds like you are saying it doesn't work that with Brocade switches and I need to somehow tell switch A that VLAN 2 exist on x port on switch B as well. Where as I'm assuming that dual-mode essential works like "switchport trunk allowed vlan all" and I'm expecting VLAN 2 to automagically carry over to switch B because I allowed it to pass on the "trunk" using this command.

It could be the suspect as looking at your running config, it should be fully operational. Heck, I even tried it out with some spare switches I had and it worked as expected but I only own ICX 7xxx switches and my main ICX does the routing.

Thank you both for trying to help. Seems I have bigger problems as I came back to a dead switch. Appears the power supply toasted on my core switch (burning smell). So I've had to put back my old Cisco switch for now. I'll explore my options for replacement after the holidays.

Really unfortunate to hear, you could look into getting a newer Brocade/Ruckus switch (e.g. ICX7150 or ICX7250) if you liked the feature set.

 

[cyinite]

just a heads up, the icx7150-24p/48p (and non-P variants!) can be had for <$200 on ebay these days, and these are arguably a contender for the "go-to" homelab switch. You can run it fanless (no need for any mods, just a simple CLI command) if you are okay with 150 watts of PoE output. They also sip power (15ish watts for non-P and 30-ish for P)

As well as make sure you reach out to sellers to double check information on the listing. eBay had a seller posting an ICX7150-24P 4X1G as the non-P varient and I confirmed the listing with them to get it for under $140 USD! Been loving how it's fanless as I've posted in the past trying to modify a 7250 to make it near silent.

 

[autoturk]

As well as make sure you reach out to sellers to double check information on the listing. eBay had a seller posting an ICX7150-24P 4X1G as the non-P varient and I confirmed the listing with them to get it for under $140 USD! Been loving how it's fanless as I've posted in the past trying to modify a 7250 to make it near silent.

Yup! Had a similar experience. Also no need for a console cable. A simple usb-c to usb-a cable works to access the console (for some reason though usb-c to usb-c doesn’t).

 

[kapone]

its ok I started reading about VRFs. New thing to learn!

the devices on the other VLANs are iPhones. Can’t dual home

ahhh...Apple...the bane of networking...

Kudos for jumping into VRFs, it's always good to know more. That said, there may be more to it, as there may be broadcasts/mDNS involved. I recall having similar issues when I was trying to put all my media devices, including an Apple TV...on a separate VLAN, and my iPhone (and the Apple TV) started screaming and shouting at me.

 

[seatrope]

ahhh...Apple...the bane of networking...

Kudos for jumping into VRFs, it's always good to know more. That said, there may be more to it, as there may be broadcasts/mDNS involved. I recall having similar issues when I was trying to put all my media devices, including an Apple TV...on a separate VLAN, and my iPhone (and the Apple TV) started screaming and shouting at me.

@kapone can you give me at least a conceptual way of how you could go about solving this issue with VRFs?

Basically i need different routing based on the source of the packet. If I implemented VRFs would my OpnSense router need to be VRF aware to handle it? as the packet is being sent back to OpnSense for Source NAT (which is really why I am doing all this routing).
OpnSense is i think sadly not VRF capable.

Thanks!!

 

[kapone]

@kapone can you give me at least a conceptual way of how you could go about solving this issue with VRFs?

Basically i need different routing based on the source of the packet. If I implemented VRFs would my OpnSense router need to be VRF aware to handle it? as the packet is being sent back to OpnSense for Source NAT (which is really why I am doing all this routing).
OpnSense is i think sadly not VRF capable.

Thanks!!

Lemme think about it (it's been a while since I played with this type of stuff).

On a different note...got the Mellanox SX6036 today, updated to the latest version...applied licenses...played with fans/PWM till I was somewhat happy...and idle power consumption is...36w.

Holy shit. This is a 36x40gb port switch!

 

[BoGs]

Lemme think about it (it's been a while since I played with this type of stuff).

On a different note...got the Mellanox SX6036 today, updated to the latest version...applied licenses...played with fans/PWM till I was somewhat happy...and idle power consumption is...36w.

Holy shit. This is a 36x40gb port switch!

have ths same one sitting in the garage for tinkering after holidays before new years. Any tips on where you got the fw and process you took? esp with the licenses.

 

[seatrope]

Lemme think about it (it's been a while since I played with this type of stuff).

On a different note...got the Mellanox SX6036 today, updated to the latest version...applied licenses...played with fans/PWM till I was somewhat happy...and idle power consumption is...36w.

Holy shit. This is a 36x40gb port switch!

exploring further @kapone would PBR (policy based routing) be able to achieve the same desired result of routing a packet based on source and destination addresses?

 

[clcorbin]

So I need some (more) help. I have been running two ICX 6610's (one in my server rack on the 2nd floor of my house and one in the garage where most of the original network cables terminated) in a stack. Multiple vlans, etc., all running great for about three years. Recently, I started seeing strange behavior on the garage switch. Specifically, while everything connected through it had intranet access, nothing had INTERNET access. This was true for IOT items, computers, etc.

Everything directly connected to the server rack switch still has internet access like always. I've rebooted the stack and the behavior persists.

No changes have been made to the router (pfSense) since before this started and the switch is stacked, so pfSense should only aftec vlans and not a specific switch member... I've attached my show run (pretty basic in general, but I did delete some of the vlan stuff as they were all pretty much the same other than name/number/ip address. FYI: I had spanning tree enabled (you can see some of the config stuff in the vlan entries), but I did disable it and the problem remains.

Current configuration:
!
ver 08.0.30uT7f3
!
stack unit 1
  module 1 icx6610-48p-poe-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
  no legacy-inline-power
  stack-port 1/2/1 1/2/6
stack unit 2
  module 1 icx6610-48p-poe-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
  no legacy-inline-power
  priority 200
  stack-port 2/2/1 2/2/6
stack enable
stack mac xxxx.xxxx.xxxx
!
global-stp
!
!
!
<snip>
vlan 10 name "Internet xxxxxxxxxxxx" by port
 tagged ethe 1/1/39 ethe 1/1/41 to 1/1/48 ethe 1/2/2 to 1/2/5 ethe 1/2/7 to 1/2/10 ethe 1/3/1 to 1/3/4 ethe 2/1/6 to 2/1/8 ethe 2/2/2 to 2/2/4 ethe 2/2/7 to 2/2/9 ethe 2/3/1 to 2/3/8
 untagged ethe 1/1/22
 router-interface ve 10
!
<snip>
!
!
!
!
lacp-cfg-det-dis
aaa authentication web-server default local
aaa authentication login default local
default-vlan-id xx
enable telnet authentication
enable aaa console
hostname xxxx
ip dhcp-client disable
ip dns server-address xxxxx xxxxx xxxxx
ip route next-hop-enable-default
ip route 0.0.0.0/0 xxxxx
!
logging buffered 100
telnet login-timeout 10
no telnet server
username xxxxx password .....
cdp run
fdp run
snmp-server community ..... ro
!
!
clock summer-time
clock timezone us xxxxx
!
!
ntp
 server 132.163.96.3
 server 132.163.96.4
 server 132.163.96.6
 server 132.163.96.2
 server 132.163.96.1
!
!
no web-management http
hitless-failover enable
!
interface ethernet 1/1/1
 port-name xxxxx
!
interface ethernet 1/1/2
 port-name xxxxx
!
interface ethernet 1/1/3
 port-name xxxxx
 spanning-tree 802-1w admin-edge-port
!
interface ethernet 1/1/4
 port-name xxxxx
 spanning-tree 802-1w admin-edge-port
!
interface ethernet 1/1/5
 port-name xxxxx
 spanning-tree 802-1w admin-edge-port
!
interface ethernet 1/1/6
 port-name xxxxx
 spanning-tree 802-1w admin-edge-port
!
interface ethernet 1/1/7
 port-name xxxxx
!
interface ethernet 1/1/8
 port-name xxxxx
!
interface ethernet 1/1/9
 port-name xxxxx
!
interface ethernet 1/1/10
 port-name xxxxx
!
interface ethernet 1/1/11
 port-name xxxxx
 spanning-tree 802-1w admin-edge-port
!
interface ethernet 1/1/12
 port-name xxxxx
 spanning-tree 802-1w admin-edge-port
!
interface ethernet 1/1/13
 port-name xxxxx
 spanning-tree 802-1w admin-edge-port
!
interface ethernet 1/1/14
 port-name xxxxx
 spanning-tree 802-1w admin-edge-port
!
interface ethernet 1/1/15
 port-name xxxxx
 spanning-tree 802-1w admin-edge-port
 inline power
!
interface ethernet 1/1/16
 port-name xxxxx
 spanning-tree 802-1w admin-edge-port
 inline power
!
interface ethernet 1/1/17
 port-name xxxxx
 spanning-tree 802-1w admin-edge-port
 inline power
!
interface ethernet 1/1/18
 port-name xxxxx
 spanning-tree 802-1w admin-edge-port
 inline power
!
interface ethernet 1/1/19
 port-name xxxxx
 spanning-tree 802-1w admin-edge-port
 inline power
!
interface ethernet 1/1/20
 port-name xxxxx
 spanning-tree 802-1w admin-edge-port
 inline power
!
interface ethernet 1/1/21
 port-name xxxxx
 spanning-tree 802-1w admin-edge-port
!
interface ethernet 1/1/22
 port-name xxxxx
!
interface ethernet 1/1/39
 port-name xxxxx
 spanning-tree 802-1w admin-pt2pt-mac
 inline power power-by-class 4
!
interface ethernet 1/1/41
 port-name xxxxx
 spanning-tree 802-1w admin-pt2pt-mac
!
interface ethernet 1/1/42
 port-name xxxxx
 inline power
!
interface ethernet 1/1/43
 port-name xxxxx
 spanning-tree 802-1w admin-pt2pt-mac
 inline power
!
interface ethernet 1/1/44
 port-name xxxxx
 inline power
!
interface ethernet 1/1/45
 port-name xxxxx
!
interface ethernet 1/1/46
 port-name xxxxx
!
interface ethernet 1/1/47
 port-name xxxxx
!
interface ethernet 1/1/48
 port-name xxxxx
!
interface ethernet 1/2/2
 port-name xxxxx
!
interface ethernet 1/2/3
 port-name xxxxx
!
interface ethernet 1/2/4
 port-name xxxxx
!
interface ethernet 1/2/5
 port-name xxxxx
!
interface ethernet 1/2/7
 port-name xxxxx
!
interface ethernet 1/2/8
 port-name xxxxx
!
interface ethernet 1/2/9
 port-name xxxxx
!
interface ethernet 1/2/10
 port-name xxxxx
!
interface ethernet 1/3/1
 port-name xxxxx
 speed-duplex 10G-full
 spanning-tree 802-1w admin-pt2pt-mac
!
interface ethernet 1/3/2
 port-name xxxxx
 speed-duplex 10G-full
!
interface ethernet 1/3/3
 port-name xxxxx
 speed-duplex 10G-full
!
interface ethernet 1/3/4
 port-name xxxxx
 speed-duplex 10G-full
!
interface ethernet 1/3/5
 port-name xxxxx
 speed-duplex 10G-full
!
interface ethernet 1/3/6
 port-name xxxxx
 speed-duplex 10G-full
!
interface ethernet 1/3/7
 port-name xxxxx
 speed-duplex 10G-full
!
interface ethernet 1/3/8
 port-name xxxxx
 speed-duplex 10G-full
 spanning-tree 802-1w admin-edge-port
!
interface ethernet 2/1/1
 port-name xxxxx
!
interface ethernet 2/1/2
 port-name xxxxx
!
interface ethernet 2/1/3
 port-name xxxxx
!
interface ethernet 2/1/4
 port-name xxxxx
!
interface ethernet 2/1/5
 port-name xxxxx
 inline power
!
interface ethernet 2/1/6
 port-name xxxxx
 inline power power-by-class 4
!
interface ethernet 2/1/7
 port-name xxxxx
 inline power power-by-class 4
!
interface ethernet 2/1/8
 port-name xxxxx
 inline power power-by-class 4
!
interface ethernet 2/1/9
 port-name xxxxx
!
interface ethernet 2/1/10
 port-name xxxxx
!
interface ethernet 2/1/11
 port-name xxxxx
 spanning-tree 802-1w admin-edge-port
!
interface ethernet 2/1/12
 port-name xxxxx
 spanning-tree 802-1w admin-edge-port
!
interface ethernet 2/1/13
 port-name xxxxx
 spanning-tree 802-1w admin-edge-port
 inline power power-by-class 4
!
interface ethernet 2/1/14
 port-name xxxxx
 spanning-tree 802-1w admin-edge-port
!
interface ethernet 2/1/15
 port-name xxxxx
 spanning-tree 802-1w admin-edge-port
!
interface ethernet 2/1/16
 port-name xxxxx
 spanning-tree 802-1w admin-edge-port
!
interface ethernet 2/1/17
 port-name xxxxx
!
interface ethernet 2/1/18
 port-name xxxxx
!
interface ethernet 2/1/19
 port-name xxxxx
!
interface ethernet 2/1/20
 port-name xxxxx
!
interface ethernet 2/1/21
 port-name xxxxx
!
interface ethernet 2/1/22
 port-name xxxxx
!
interface ethernet 2/1/23
 port-name xxxxx
!
interface ethernet 2/1/24
 port-name xxxxx
!
interface ethernet 2/1/25
 port-name xxxxx
!
interface ethernet 2/1/26
 port-name xxxxx
!
interface ethernet 2/1/27
 port-name xxxxx
 inline power
!
interface ethernet 2/1/28
 port-name xxxxx
 inline power
!
interface ethernet 2/1/29
 port-name xxxxx
 inline power
!
interface ethernet 2/1/30
 port-name xxxxx
 inline power
!
interface ethernet 2/1/31
 port-name xxxxx
 inline power
!
interface ethernet 2/1/32
 port-name xxxxx
 inline power
!
interface ethernet 2/1/33
 port-name xxxxx
 inline power
!
interface ethernet 2/1/34
 port-name xxxxx
 inline power
!
interface ethernet 2/1/35
 port-name xxxxx
!
interface ethernet 2/1/36
 port-name xxxxx
!
interface ethernet 2/1/37
 port-name xxxxx
!
interface ethernet 2/1/38
 port-name xxxxx
!
interface ethernet 2/1/39
 port-name xxxxx
!
interface ethernet 2/1/40
 port-name xxxxx
!
interface ethernet 2/1/41
 port-name xxxxx
!
interface ethernet 2/1/42
 port-name xxxxx
!
interface ethernet 2/1/43
 port-name xxxxx
!
interface ethernet 2/1/44
 port-name xxxxx
 inline power power-by-class 4
!
interface ethernet 2/1/45
 port-name xxxxx
 spanning-tree 802-1w admin-edge-port
!
interface ethernet 2/1/46
 port-name xxxxx
 spanning-tree 802-1w admin-edge-port
!
interface ethernet 2/1/47
 port-name xxxxx
 spanning-tree 802-1w admin-edge-port
!
interface ethernet 2/1/48
 port-name xxxxx
 spanning-tree 802-1w admin-edge-port
!
interface ethernet 2/2/2
 port-name xxxxx
 spanning-tree 802-1w admin-edge-port
!
interface ethernet 2/2/3
 port-name xxxxx
 spanning-tree 802-1w admin-edge-port
!
interface ethernet 2/2/4
 port-name xxxxx
 spanning-tree 802-1w admin-edge-port
!
interface ethernet 2/2/5
 port-name xxxxx
 spanning-tree 802-1w admin-edge-port
!
interface ethernet 2/2/7
 port-name xxxxx
 spanning-tree 802-1w admin-edge-port
!
interface ethernet 2/2/8
 port-name xxxxx
 spanning-tree 802-1w admin-edge-port
!
interface ethernet 2/2/9
 port-name xxxxx
 spanning-tree 802-1w admin-edge-port
!
interface ethernet 2/2/10
 port-name xxxxx
 spanning-tree 802-1w admin-edge-port
!
interface ethernet 2/3/1
 port-name xxxxx
 speed-duplex 10G-full
 spanning-tree 802-1w admin-edge-port
!
interface ethernet 2/3/2
 port-name xxxxx
 speed-duplex 10G-full
 spanning-tree 802-1w admin-edge-port
!
interface ethernet 2/3/3
 port-name xxxxx
 speed-duplex 10G-full
 spanning-tree 802-1w admin-edge-port
!
interface ethernet 2/3/4
 port-name xxxxx
 speed-duplex 10G-full
 spanning-tree 802-1w admin-edge-port
!
interface ethernet 2/3/5
 port-name xxxxx
 speed-duplex 10G-full
!
interface ethernet 2/3/6
 port-name xxxxx
 disable
 speed-duplex 10G-full
!
interface ethernet 2/3/7
 port-name xxxxx
 disable
 speed-duplex 10G-full
!
interface ethernet 2/3/8
 port-name xxxxx
 speed-duplex 10G-full
!
interface ve 1
 ip address xxxxx xxxxx
 ip helper-address 1 xxxxx
 ip helper-address 2 xxxx
!
interface ve 2
 ip address xxxxx xxxxx
!
interface ve 10
 ip address xxxxx xxxxx
 ip helper-address 1 xxxxxxxxxxxx
 ip helper-address 2 xxxxxxxxxxxx
 !
<snip>
!
!
lldp tagged-packets process
lldp run
!
!
ip ssh  idle-time 240
ip ssh  interactive-authentication no
!
!
end

 

[cyinite]

exploring further @kapone would PBR (policy based routing) be able to achieve the same desired result of routing a packet based on source and destination addresses?

Looking through the FastIron security guide, PBR might be exactly what they are looking for. You set an ACL of what is permitted through the policy (source and destination) and add that ACL to a route map applied to a port or VE that points to the router (pf/OPNsense) where you can set source NAT. Below is the example configuration from the chapter.

device# configure terminal
device(config)# ip access-list standard 99
device(config-std-ipacl-99)# permit 10.157.23.0 0.0.0.255
device(config-std-ipacl-99)# exit
device(config)# route-map test-route permit 99
device(config-routemap test-route)# match ip address 99
device(config-routemap test-route)# set ip next-hop 192.168.3.1
device(config-routemap test-route)# exit
device(config)# interface ethernet 1/1/3
device(config-if-e1000-1/1/3)# ip policy route-map test-route
device(config-if-e1000-1/1/3)# end

 

[molnart]

how to configure the brocade switch to pass any VLAN on the default VLAN1 ? i am defining the VLANs on my OPNsense router, but it looks like on the brocade i need to explicitly mark the VLANs for any port. how do I achieve that the Brocade immediately picks up and passes a newly created VLAN from the OPNsense router?

 

[BoGs]

how to configure the brocade switch to pass any VLAN on the default VLAN1 ? i am defining the VLANs on my OPNsense router, but it looks like on the brocade i need to explicitly mark the VLANs for any port. how do I achieve that the Brocade immediately picks up and passes a newly created VLAN from the OPNsense router?

You need to define the VLANs (any newly created ones) and tag the port to the OPNsense. There is no detection of new vlans and automagic that I am aware of in Brocade. Arista and Cisco have a trunk configuration but its better to just define them so vlans do not run everywhere in case of comprimise. Also it does not pass vlans on any vlans, it passes the frames over the port. VLAN 1 is just the default vlan on all ports.

 

[Avamander]

Hmm, I seem to have locked myself out of my ICX 7250 by accident, and all of the ports on the switch seem closed/filtered so I can't access SSH or the web interface any more. I have a Brocade console cable but there's absolutely no output (and I can't recall it ever working, so it might be burnt). I have not configured the management interface AFAIK (unless there are defaults I am not aware of).

Holding the reset button for whatever time period has had no effect, I still see it using the previous/broken config.

Is there a way to factory reset or restore access (without working console)?

 

[BoGs]

Hmm, I seem to have locked myself out of my ICX 7250 by accident, and all of the ports on the switch seem closed/filtered so I can't access SSH or the web interface any more. I have a Brocade console cable but there's absolutely no output (and I can't recall it ever working, so it might be burnt). I have not configured the management interface AFAIK (unless there are defaults I am not aware of).

Holding the reset button for whatever time period has had no effect, I still see it using the previous/broken config.

Is there a way to factory reset or restore access (without working console)?

If I remember correctly if you look at fohdeshas guide you can see how to console and get in the boot prompt `b` and do factory reset and go through the setup. You should be able to console connect though, and see the output if you press enter to login with admin. Try connecting console and pull power and plug back in do you see the switch booting up? I do not think I have ever locked myself out of my switch if you did default setup.