Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[challaday]

First of all, thanks for all the advice and setup walkthroughs, they've been really helpful as I jump back into networking after being away for a long time. I purchased a ICX6450-48 (non-PoE) and thought the fan was a bit loud, despite very low load and very low temps (33-34C). The fan is set to auto, and shows that it is running at speed 1.

I switched out the fan to a Sunon fan that I've seen in a number of online guides as being nearly silent, but the 6450 is still the loudest item in the rack, including the five Dell R630 servers. To me, it seems like the fan is just not throttling back.

The version of FastIron I'm running (08.0.30) doesn't seem to have a fan-speed command, so is there something I'm missing to configure the fan?

 

[NablaSquaredG]

To me, it seems like the fan is just not throttling back.

Check whether the FAN Control MOSFET is fried

Maybe we should add that to the start post @fohdeesha ?

 

[kemic]

Forgive me if this has already been discussed.

I have 4 ICX6610s in a stack using the 40Gbe ports on the back (not the 4x10Gbe break out ports). From the official data sheet and first post on this thread, it's mentioned that all of the rear ports are for stacking (which is of course not true). I'm curious about setting up some 40Gbe connections from my servers to the stack and have some questions:

1. Can I move the cables from the 40Gbe ports to the breakout ports and reconfigure the stack to use the breakout ports only? Thus freeing up the 40Gbe ports for connections to my servers.
2. If yes on #1, does this need any special cable?
3. If yes on #1, does this negatively impact stack performance at all? EX: 40Gbe system on switch4 talking to 40Gbe system on switch1, traffic would be traveling over the 4x10Gbe stack ports. With out knowing any better, I would assume this traffic would max out at 1x10Gbe... The only way I'd get the full 40Gbe is if both servers are plugged into the same switch in the stack. Am I off on that assumption?

Thanks!

 

[NablaSquaredG]

Can I move the cables from the 40Gbe ports to the breakout ports and reconfigure the stack to use the breakout ports only? Thus freeing up the 40Gbe ports for connections to my servers.

If you find a way, let me know. I've tried multiple times and always failed.

 

[sic0048]

Forgive me if this has already been discussed.

I have 4 ICX6610s in a stack using the 40Gbe ports on the back (not the 4x10Gbe break out ports). From the official data sheet and first post on this thread, it's mentioned that all of the rear ports are for stacking (which is of course not true). I'm curious about setting up some 40Gbe connections from my servers to the stack and have some questions:

1. Can I move the cables from the 40Gbe ports to the breakout ports and reconfigure the stack to use the breakout ports only? Thus freeing up the 40Gbe ports for connections to my servers.
2. If yes on #1, does this need any special cable?
3. If yes on #1, does this negatively impact stack performance at all? EX: 40Gbe system on switch4 talking to 40Gbe system on switch1, traffic would be traveling over the 4x10Gbe stack ports. With out knowing any better, I would assume this traffic would max out at 1x10Gbe... The only way I'd get the full 40Gbe is if both servers are plugged into the same switch in the stack. Am I off on that assumption?

Thanks!

I'm no expert, but I believe in order to get a "ring" configuration which allows for redundancy in case one or more switches goes down, you are going to need to use all four of the QSFP ports on the back of each switch. If you don't use all four, if one of the switches fails, you'll loose access to everything past that switch because there is no redundancy built in. With four switches in your system, I personally would want to make sure there is redundancy built into the design.

Ruckus has some excellent documentation on stacking. If you haven't read this guide, I would suggest that you and read it. FastIron 08.0.30 Stacking Configuration Guide | Technical Documents | Ruckus Wireless Support

EDIT - actually after looking at the guide a little more, it does seem that you can create a redundant stack only using half of the ports on the back, but it is not suggested by Ruckus. You'll also miss out on the load balancing that using all the ports will allow for. Again, with four switches in your system, I would think this stacking design needs to be robust and have as much bandwidth as possible. Using all the ports for the stack is the best way to accomplish this.

 

[dehudson]

I'm certain this has been asked previously, but I searched everything I could think of and scrolled through hundreds of pages of this thread without any luck.

I purchased an ICX 6610-24P last week, and tried to follow the information on the first page of this thread to update the firmware first.

It has no configuration at all (fyi).
Got to the boot menu without trouble.
Got an IP assigned without trouble.
Got a tftp server set up without trouble.
.....When I connect my tftp server to any of the 24 RJ45 ports... nothing. Tried two different PC's, and all 24 ports on the switch. No link, no light.

Did I miss some instructions? Do I need a base config on the switch first? Do I need to be using the OOB port for the tftp 'server'?

I appreciate any hints you can give regarding what important step(s) I missed.

 

[NablaSquaredG]

Did I miss some instructions? Do I need a base config on the switch first? Do I need to be using the OOB port for the tftp 'server'?

I appreciate any hints you can give regarding what important step(s) I missed.

I quote:

Connect to the serial/console port using a program like Putty (9600 8N1), then connect the management ethernet port to your network (do NOT use any of the regular switch ports yet).

right at the top of the guide

 

[hmw]

1. What is the transit VLAN method? Can you link to an example or elaborate? Sounds more complicated than the second method.

- create a vlan for transit traffic, attach to a port and set an IP (with a /30 subnet mask)

# vlan 253
# untagged ethernet 1/1/48
# router-interface ve 253
# ip address 10.0.253.1/30

- attach this port to an interface on OPNsense and set that interface to 10.0.253.2

- next set route on the switch

# ip route 0.0.0.0/0 10.0.253.2

And then set static routes on OPNsense back to the appropriate networks + NAT + firewall rules

2. I have configured the "route-only" method in the past and it works very well. Especially because you now have a GUI for managing DHCP/DNS and can more easily create static DHCP mappings.

- so this would be

# vlan 253
# untagged ethe 1/1/48
# int ethe 1/1/48
# route-only
# ip address 10.0.253.1/30

- And then the same stuff with OPNsense.

My question is - what is the difference between these two methods and is one preferable over the other?

 

[Hrast]

Check whether the FAN Control MOSFET is fried

I've got a ICX6450-48 that the fan doesn't spin down on, where should I be looking? Already verified that I'm at the recommended bootroom/switch code.

 

[dehudson]

I have a single ICX6610. I'm setting it up for the first time, but am confused why my config thinks I have two stack units, and what to do to correct it.
I tried the 'stack unconfigure me', but when I then type 'stack unconfigure clean' I get 'This command is not available on standalone or Active Controller'. ...no joy there.

I'm attaching some screen shots which I hope will help better explain what I'm experiencing.

Any suggestions on how to correct this?

 

[u4096]

- create a vlan for transit traffic, attach to a port and set an IP (with a /30 subnet mask)

# vlan 253
# untagged ethernet 1/1/48
# router-interface ve 253
# ip address 10.0.253.1/30

- attach this port to an interface on OPNsense and set that interface to 10.0.253.2

- next set route on the switch

# ip route 0.0.0.0/0 10.0.253.2

And then set static routes on OPNsense back to the appropriate networks + NAT + firewall rules

My question is - what is the difference between these two methods and is one preferable over the other?

Thank you for going into detail. Learned something new. Sounds like a variation then of my response earlier.

Based on @LodeRunner 's response, sounds to me that the transit VLAN offers more granularity if you are using inter-VLAN ACLs.

The way I see it is that the "plain" interface routing one would not allow VLAN-to-VLAN ACLs since the default route interface is not a VLAN. I think you can still insert rules from the specific VLANs to prohibit traffic to this plain interface and egress to OPNSense. But there is a chance of traffic "leaking" to other VLANs, within the switch. So you have to be a lot more explicit with permit/deny statements.

The transit VLAN method sounds to me cleaner as it it can simply control traffic at the VLAN level. Now I wonder what best practice is.

@LodeRunner: Would that be a fair understanding of what you wrote? When would you use one vs. the other?

 

[kpfleming]

I'm not really sure that 'best practice' applies here: the controlling factor will be whether you want *all* routing to be done by the attached router, or whether you will allow some of the routing to be done by the L3 switch. If you have more inter-VLAN traffic than the link to the attached router can handle, then you have to allow the switch to do some of the inter-VLAN routing or the traffic will be bottlenecked by that link. The same is true if the attached router doesn't have sufficient horsepower to handle the routing, even if the link is wide enough.

I recently converted from mixed-mode (some routing on the ICX stack, the rest on an attached router) to attached-router mode (with the ICX stack not using any L3 features). Doing this required upgrading the ICX-to-router link to 10Gb so that I could be sure that it would not be a bottleneck for any of my traffic. The result is that I can manage all traffic rules (ACLs) in one place, using a tool (nftables) that is more capable and that I understand better than the ICX's ACLs.

 

[hmw]

I recently converted from mixed-mode (some routing on the ICX stack, the rest on an attached router) to attached-router mode (with the ICX stack not using any L3 features). Doing this required upgrading the ICX-to-router link to 10Gb so that I could be sure that it would not be a bottleneck for any of my traffic. The result is that I can manage all traffic rules (ACLs) in one place, using a tool (nftables) that is more capable and that I understand better than the ICX's ACLs.

I had OPNsense connected via 10Gbit links to my Unifi XG24 switch and my homelab ICX6610. The Ubiquiti switches (even the so-called 'enterprise' ones) *need* a transit VLAN (hard coded to 4040) to a router and hence I had a 4040 VLAN to OPNsense carrying all the other VLANs. I got t working - however OPNsense really *hates* trunked VLANs. It would flap interfaces regardless of the permutations of VLAN hw filtering / IDPS mode / whatever other weird tunable param blah blah. At first I thought it was the XG24 - afterwards I realized it was OPNsense itself. That's when I just got hold of a ICX7650 and used that to do all the VLANs on the switch itself + NAT rules on OPNsense.

My biggest problem currently is DNS/DHCP - I am using Wimdows Server Core and it is a major pain compared to handling DHCP/DNS on OPNsense. For whatever reason Windows DHCP/DNS server will happily let clients override DHCP assigned hostnames (Unexpected DNS record registration behavior when the DHCP server manages dynamic DNS updates - Windows Server) and to change the behavior is near impossible unless you set GPO policy (impossible on non-Windows clients). Also it wont properly set search domains or domain options on MacOS clients. And it is painful to see how it treats IoT devices that might be using an older Linux stack

I think Fohdeesha actually offered money to the OPNsense developers to get them to invest in a better DHCP interface but they refused on the grounds of the code being too monolithic for any real refactoring. Several folks have also asked the Pi-Hole folks to put in a GUI for VLANs and subnets for DHCP but the Pi-Hole developers have said that it isn't the main focus and hence they won't even consider it

At this point, I am looking at using Webmin to handle ISC-DHCP and creating a basic VLAN/subnet aware GUI for PowerDNS (or both)

 

[TonyArrr]

At this point, I am looking at using Webmin to handle ISC-DHCP and creating a basic VLAN/subnet aware GUI for PowerDNS (or both)

Might be a a good time to start reading up on ISC-Kea, which is replacing ISC-DHCPd
I had been getting really into how to configure and admin ISC-DHCPd and then found out that is it actually deprecated in favour of Kea, so starting again with it before my network moves over.

 

[Nehalem501]

Hello,

I've recently acquired an ICX 6450-48 (non PoE version) and its (single) stock fan is very loud.
When reading this thread, I understand the fan should slow down after a while after the unit has finished booting, but mine seems stuck at full speed all the time without anything connected to the switch (I kept it on for over an hour once to check if the fan would slow down after some time).

I've already followed the guide to update the firmware to the latest version (08.0.30uT313).

The temperature displayed when running show chassis is always between 25° C and 40° C and for the fan it says Fan ok, speed (auto): [[1]]<->2

I don't know if it is supposed to work like that, but the fan blows air from inside the switch to outside the switch.

Is there something wrong with my unit, or should I just change the fan? (which one would be the quietest? I would prefer to keep some form of cooling rather than running it completely fanless, summers in France can get pretty hot and most homes don't have AC here)

Thanks for your help.

Yes, Fan MOSFET fried. Seems to be a common issues with those switches.

I’ve looked into this for the past few days and the issue looks to be more complicated than it seems.

I found the fan MOSFET, and the Op Amp whose output goes into the MOSFET. Both being quite close to the fan connector the PCB.
With a multimeter and an oscilloscope I measured the voltage and looked at the signal on all pins and everything is absolutely the same wether the fan speed is on [1] or [2]. At least it confirms that the fan runs at the max speed all the time (and I have indeed measured 12V going into the fan).

I’ve tried to see where the trace that goes into the input of the op amp leads. But it goes to a via and then the trace continues inone of the internal layers of the PCB, so next to impossible to find out without some schematics (which I couldn’t find on the internet).

For the time being I have disconnected the fan, and reassembled the unit. After keeping the switch on for a few hours, with some traffic going through it (watching a 4K Netflix stream) the temperatures were around 65°C, which seems quite high.

Which fan would you recommend that would provide ok cooling while being silent when running with a 12V input ? Thanks.

 

[zeroturnpete]

Thanks for all of the useful info, just picked up a 6610 (non poe) for 70 bucks. What is the best way to go about getting 10 gig licenses around the forum

 

[hmw]

Might be a a good time to start reading up on ISC-Kea, which is replacing ISC-DHCPd
I had been getting really into how to configure and admin ISC-DHCPd and then found out that is it actually deprecated in favour of Kea, so starting again with it before my network moves over.

Yeah I have been reading up into Kea vs DHCPd. On the face of it - Kea sounds like a great alternative. However, the reason you don't see widespread adoption of Kea amongst open source projects is because:

[1] most of the features that make Kea stand out over and above DHCPd like the API for subnets that allows adding and deleting subnets without restarting the Kea server - are hidden behind a paywall, with the cheapest option being $550 for a 1 year subscription + 5 years usage assuming you have < 1000 active leases

[2] You can run Kea with a 'memfile' backend but some of the advanced options need a SQL database like MySQL or PostgreSQL. And Kea doesn't do SQLite

Interestingly - on the OPNsense forums, they were floating Free Radius as a DHCP alternative, from ver 3.0.x it has a full DHCP stack and there's some mods to let it use SQLite (so that host leases etc are inside a database), might just have a look at that ...

 

[custom90gt]

Thanks for all of the useful info, just picked up a 6610 (non poe) for 70 bucks. What is the best way to go about getting 10 gig licenses around the forum

by reading the first post and following those instructions.

 

[TonyArrr]

are hidden behind a paywall, with the cheapest option being $550 for a 1 year subscription + 5 years usage assuming you have < 1000 active leases

Whoa, somehow I completely missed that. Very steer clear then!

 

[Mushishi]

https://www.fs.com/products/69905.html

Okay, cool, I see the "dual-compatibility solutions" allows me to pick different ends. But I was also hoping for some advice as to whether the oracle-branded intel card takes an Intel module or something else, and how I might discover that (short of "plug it in, it doesn't work"). Or, if the Brocade isn't restricted, I just go with an Intel-to-Intel connection?
I guess I should pick up some fiber too, to share shipping costs. And a few other things that might come up as I test things out. Ah, discovery!

I got one of the Oracle branded x520's some time ago also and while i can't remember if i had any problems with the optics i know i followed this post to unlock the card to accept all optics.

https://forums.servethehome.com/index.php?threads/patching-intel-x520-eeprom-to-unlock-all-sfp-transceivers.24634/