Did you add a 60/80mm super slim (10mm) fan on the heatsink of the asic already?
Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)
- Thread starter fohdeesha
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
Trying to setup OPNsense with the ICX7650 and VLANs, will have a separate Windows DNS/DHCP server for DHCP + make the OPNsense into a gateway with static routes back to the 7650.
I was reading about possible configurations and one is to use a transit VLAN between OPNsense and the L3 switch and another uses a routed port with an IP address set on that port ( so "route-only" )
Is there a reason to choose one method over the other?
I was reading about possible configurations and one is to use a transit VLAN between OPNsense and the L3 switch and another uses a routed port with an IP address set on that port ( so "route-only" )
Is there a reason to choose one method over the other?
I am getting "Failed to add ICX [192.168.x.xxx]. The ICX already exists. Please enter a different one." in my Unleashed (200.14.6.1.179) logs.
I recently upgraded my ICX7150-c12p to 09.0.10e from 08.0.95f and I may have inadvertently removed the switch in Unleashed then re-added it using the same IP. Anyone know how can I force Unleashed to stop trying to add the switch?
EDIT: problem solved by just restarting Unleashed...
I recently upgraded my ICX7150-c12p to 09.0.10e from 08.0.95f and I may have inadvertently removed the switch in Unleashed then re-added it using the same IP. Anyone know how can I force Unleashed to stop trying to add the switch?
EDIT: problem solved by just restarting Unleashed...
Last edited:
1. What is the transit VLAN method? Can you link to an example or elaborate? Sounds more complicated than the second method.
2. I have configured the "route-only" method in the past and it works very well. Especially because you now have a GUI for managing DHCP/DNS and can more easily create static DHCP mappings.
Like you described it's pretty simple. You just to have to setup the routes on OPNSense for all the VLANs you want to transit/egress to internet and configured a helper address for each corresponding VLAN in the 7650 config. Something like:
Code:
interface ve 3
port-name LAN
ip address 192.168.3.3 255.255.255.0
ip helper-address 1 192.168.1.1
.Transit VLAN is where you let the core switch handle all inter-VLAN routing and it has a default gateway on a separate VLAN that's a point to point to the firewall. The firewall has static routes for each subnet back via the switch's IP on the transit VLAN.
Here's a very barebones config I did for another post to demonstrate:
Code:
ICX7150-C12 Router#sh run
Current configuration:
!
ver 08.0.95eT213
!
stack unit 1
module 1 icx7150-c12-poe-port-management-module
module 2 icx7150-2-copper-port-2g-module
module 3 icx7150-2-sfp-plus-port-20g-module
stack-port 1/3/1
stack-port 1/3/2
!
global-stp
vlan 1 name DEFAULT-VLAN by port
spanning-tree
!
vlan 11 by port
tagged ethe 1/1/1
untagged ethe 1/1/11
router-interface ve 11
!
vlan 12 by port
tagged ethe 1/1/1
untagged ethe 1/1/3
router-interface ve 12
!
vlan 20 by port
tagged ethe 1/1/1
router-interface ve 20
!
ip dhcp-client disable
ip route 0.0.0.0/0 172.16.21.2
!
!
interface ve 11
ip address 10.100.11.1 255.255.255.0
!
interface ve 12
ip address 10.100.12.1 255.255.255.0
ip helper-address 1 10.100.11.2
!
interface ve 20
ip address 172.16.21.1 255.255.255.0
endSwitch IP and route tables:
Code:
ICX7150-C12 Router#sh ip add
IP Address Type Lease Time Interface
10.100.11.1 Static N/A ve11
10.100.12.1 Static N/A ve12
172.16.21.1 Static N/A ve20
ICX7150-C12 Router#sh ip rout
Total number of IP routes: 4
Type Codes - B:BGP D:Connected O:OSPF R:RIP S:Static; Cost - Dist/Metric
BGP Codes - i:iBGP e:eBGP
OSPF Codes - i:Inter Area 1:External Type 1 2:External Type 2
STATIC Codes - v:Inter-VRF
Destination Gateway Port Cost Type Uptime
1 0.0.0.0/0 172.16.21.2 ve 20 1/1 S 18m57s
2 10.100.11.0/24 DIRECT ve 11 0/0 D 26m26s
3 10.100.12.0/24 DIRECT ve 12 0/0 D 28m5s
4 172.16.21.0/24 DIRECT ve 20 0/0 D 19m28s
ICX7150-C12 Router#pfSense gateway and routes:
View attachment 21045
View attachment 21046
Windows VM in VLAN 11, IP 10.100.11.2/24, running DHCP server, showing client with IP 10.100.12.2/24:
View attachment 21047
Works well if your inter-VLAN security needs can be handled with ACLs on the switch (or if you just want physical separation for your subnets.
First of all, thanks for all the advice and setup walkthroughs, they've been really helpful as I jump back into networking after being away for a long time. I purchased a ICX6450-48 (non-PoE) and thought the fan was a bit loud, despite very low load and very low temps (33-34C). The fan is set to auto, and shows that it is running at speed 1.
I switched out the fan to a Sunon fan that I've seen in a number of online guides as being nearly silent, but the 6450 is still the loudest item in the rack, including the five Dell R630 servers. To me, it seems like the fan is just not throttling back.
The version of FastIron I'm running (08.0.30) doesn't seem to have a fan-speed command, so is there something I'm missing to configure the fan?
I switched out the fan to a Sunon fan that I've seen in a number of online guides as being nearly silent, but the 6450 is still the loudest item in the rack, including the five Dell R630 servers. To me, it seems like the fan is just not throttling back.
The version of FastIron I'm running (08.0.30) doesn't seem to have a fan-speed command, so is there something I'm missing to configure the fan?
Check whether the FAN Control MOSFET is fried
Maybe we should add that to the start post @fohdeesha ?
Forgive me if this has already been discussed.
I have 4 ICX6610s in a stack using the 40Gbe ports on the back (not the 4x10Gbe break out ports). From the official data sheet and first post on this thread, it's mentioned that all of the rear ports are for stacking (which is of course not true). I'm curious about setting up some 40Gbe connections from my servers to the stack and have some questions:
I have 4 ICX6610s in a stack using the 40Gbe ports on the back (not the 4x10Gbe break out ports). From the official data sheet and first post on this thread, it's mentioned that all of the rear ports are for stacking (which is of course not true). I'm curious about setting up some 40Gbe connections from my servers to the stack and have some questions:
- Can I move the cables from the 40Gbe ports to the breakout ports and reconfigure the stack to use the breakout ports only? Thus freeing up the 40Gbe ports for connections to my servers.
- If yes on #1, does this need any special cable?
- If yes on #1, does this negatively impact stack performance at all? EX: 40Gbe system on switch4 talking to 40Gbe system on switch1, traffic would be traveling over the 4x10Gbe stack ports. With out knowing any better, I would assume this traffic would max out at 1x10Gbe... The only way I'd get the full 40Gbe is if both servers are plugged into the same switch in the stack. Am I off on that assumption?
I'm no expert, but I believe in order to get a "ring" configuration which allows for redundancy in case one or more switches goes down, you are going to need to use all four of the QSFP ports on the back of each switch. If you don't use all four, if one of the switches fails, you'll loose access to everything past that switch because there is no redundancy built in. With four switches in your system, I personally would want to make sure there is redundancy built into the design.
Ruckus has some excellent documentation on stacking. If you haven't read this guide, I would suggest that you and read it. FastIron 08.0.30 Stacking Configuration Guide | Technical Documents | Ruckus Wireless Support
EDIT - actually after looking at the guide a little more, it does seem that you can create a redundant stack only using half of the ports on the back, but it is not suggested by Ruckus. You'll also miss out on the load balancing that using all the ports will allow for. Again, with four switches in your system, I would think this stacking design needs to be robust and have as much bandwidth as possible. Using all the ports for the stack is the best way to accomplish this.
Last edited:
I'm certain this has been asked previously, but I searched everything I could think of and scrolled through hundreds of pages of this thread without any luck.
I purchased an ICX 6610-24P last week, and tried to follow the information on the first page of this thread to update the firmware first.
It has no configuration at all (fyi).
Got to the boot menu without trouble.
Got an IP assigned without trouble.
Got a tftp server set up without trouble.
.....When I connect my tftp server to any of the 24 RJ45 ports... nothing. Tried two different PC's, and all 24 ports on the switch. No link, no light.
Did I miss some instructions? Do I need a base config on the switch first? Do I need to be using the OOB port for the tftp 'server'?
I appreciate any hints you can give regarding what important step(s) I missed.
I purchased an ICX 6610-24P last week, and tried to follow the information on the first page of this thread to update the firmware first.
It has no configuration at all (fyi).
Got to the boot menu without trouble.
Got an IP assigned without trouble.
Got a tftp server set up without trouble.
.....When I connect my tftp server to any of the 24 RJ45 ports... nothing. Tried two different PC's, and all 24 ports on the switch. No link, no light.
Did I miss some instructions? Do I need a base config on the switch first? Do I need to be using the OOB port for the tftp 'server'?
I appreciate any hints you can give regarding what important step(s) I missed.
- create a vlan for transit traffic, attach to a port and set an IP (with a /30 subnet mask)
Code:
# vlan 253
# untagged ethernet 1/1/48
# router-interface ve 253
# ip address 10.0.253.1/30- next set route on the switch
Code:
# ip route 0.0.0.0/0 10.0.253.2- so this would be
Code:
# vlan 253
# untagged ethe 1/1/48
# int ethe 1/1/48
# route-only
# ip address 10.0.253.1/30My question is - what is the difference between these two methods and is one preferable over the other?
I've got a ICX6450-48 that the fan doesn't spin down on, where should I be looking? Already verified that I'm at the recommended bootroom/switch code.
I have a single ICX6610. I'm setting it up for the first time, but am confused why my config thinks I have two stack units, and what to do to correct it.
I tried the 'stack unconfigure me', but when I then type 'stack unconfigure clean' I get 'This command is not available on standalone or Active Controller'. ...no joy there.
I'm attaching some screen shots which I hope will help better explain what I'm experiencing.
Any suggestions on how to correct this?
I tried the 'stack unconfigure me', but when I then type 'stack unconfigure clean' I get 'This command is not available on standalone or Active Controller'. ...no joy there.
I'm attaching some screen shots which I hope will help better explain what I'm experiencing.
Any suggestions on how to correct this?
Attachments
Thank you for going into detail. Learned something new. Sounds like a variation then of my response earlier.
Based on @LodeRunner 's response, sounds to me that the transit VLAN offers more granularity if you are using inter-VLAN ACLs.
The way I see it is that the "plain" interface routing one would not allow VLAN-to-VLAN ACLs since the default route interface is not a VLAN. I think you can still insert rules from the specific VLANs to prohibit traffic to this plain interface and egress to OPNSense. But there is a chance of traffic "leaking" to other VLANs, within the switch. So you have to be a lot more explicit with permit/deny statements.
The transit VLAN method sounds to me cleaner as it it can simply control traffic at the VLAN level. Now I wonder what best practice is.
@LodeRunner: Would that be a fair understanding of what you wrote? When would you use one vs. the other?
I'm not really sure that 'best practice' applies here: the controlling factor will be whether you want *all* routing to be done by the attached router, or whether you will allow some of the routing to be done by the L3 switch. If you have more inter-VLAN traffic than the link to the attached router can handle, then you have to allow the switch to do some of the inter-VLAN routing or the traffic will be bottlenecked by that link. The same is true if the attached router doesn't have sufficient horsepower to handle the routing, even if the link is wide enough.
I recently converted from mixed-mode (some routing on the ICX stack, the rest on an attached router) to attached-router mode (with the ICX stack not using any L3 features). Doing this required upgrading the ICX-to-router link to 10Gb so that I could be sure that it would not be a bottleneck for any of my traffic. The result is that I can manage all traffic rules (ACLs) in one place, using a tool (nftables) that is more capable and that I understand better than the ICX's ACLs.
I recently converted from mixed-mode (some routing on the ICX stack, the rest on an attached router) to attached-router mode (with the ICX stack not using any L3 features). Doing this required upgrading the ICX-to-router link to 10Gb so that I could be sure that it would not be a bottleneck for any of my traffic. The result is that I can manage all traffic rules (ACLs) in one place, using a tool (nftables) that is more capable and that I understand better than the ICX's ACLs.
I had OPNsense connected via 10Gbit links to my Unifi XG24 switch and my homelab ICX6610. The Ubiquiti switches (even the so-called 'enterprise' ones) *need* a transit VLAN (hard coded to 4040) to a router and hence I had a 4040 VLAN to OPNsense carrying all the other VLANs. I got t working - however OPNsense really *hates* trunked VLANs. It would flap interfaces regardless of the permutations of VLAN hw filtering / IDPS mode / whatever other weird tunable param blah blah. At first I thought it was the XG24 - afterwards I realized it was OPNsense itself. That's when I just got hold of a ICX7650 and used that to do all the VLANs on the switch itself + NAT rules on OPNsense.
My biggest problem currently is DNS/DHCP - I am using Wimdows Server Core and it is a major pain compared to handling DHCP/DNS on OPNsense. For whatever reason Windows DHCP/DNS server will happily let clients override DHCP assigned hostnames (Unexpected DNS record registration behavior when the DHCP server manages dynamic DNS updates - Windows Server) and to change the behavior is near impossible unless you set GPO policy (impossible on non-Windows clients). Also it wont properly set search domains or domain options on MacOS clients. And it is painful to see how it treats IoT devices that might be using an older Linux stack
I think Fohdeesha actually offered money to the OPNsense developers to get them to invest in a better DHCP interface but they refused on the grounds of the code being too monolithic for any real refactoring. Several folks have also asked the Pi-Hole folks to put in a GUI for VLANs and subnets for DHCP but the Pi-Hole developers have said that it isn't the main focus and hence they won't even consider it
At this point, I am looking at using Webmin to handle ISC-DHCP and creating a basic VLAN/subnet aware GUI for PowerDNS (or both)