Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[Robert Bakley]

I just stacked a pair of 6450-48P's using Terry Henry's YouTube guide and I discovered something peculiar. My 6450's were used and I was testing them out prior to replacing my main switches (a Quanta LB4M and a Cisco SG300-28MPP). One of the 6450's was trunked to my LB4M. When i issued the stack enable command the Quanta ceased to transfer packets and I lost the rest of my network. Pulling the trunk cable restored the communication on the LB4M. Not sure why. I would think that this command would only affect other Brocade switches and then only the same series of switches (64xx, 72xx, etc.) Is this because the Quanta software is a close clone or cousin of the FastIron firmware? I think I recall the Quanta can run FastIron software. It is currently running Vxxworks 5.5.1 Sw 1.0.2.17.

Anyway, a heads up to anyone trying this in the future.

 

[tubs-ffm]

3) Do the 7250-24 and 7250-48 *both* idle at 50w, while the 6450-24 idles at 25w? If so, it might be worth me giving up the extra 10G SFP ports for my use case; otherwise I'll probably go 7250

If it is about power consumption you also can have a look on the 7150-24. Here you get low power consumption and additionally fanless if you concern about noise.
brochure-icx-7150.pdf (ruckussecurity.com)

 

[NickM]

If it is about power consumption you also can have a look on the 7150-24. Here you get low power consumption and additionally fanless if you concern about noise.
brochure-icx-7150.pdf (ruckussecurity.com)

It's heat and noise rather than the energy cost that are my motivation here, but they're obviously directly linked.

That's a good idea, though it looks like the cheapest 7150-24 I see is still $400ish shipped. I'll keep an eye out, but for double+ the cost of the other options I'll just deal with some noise

 

[tubs-ffm]

It's heat and noise rather than the energy cost that are my motivation here, but they're obviously directly linked.

That's a good idea, though it looks like the cheapest 7150-24 I see is still $400ish shipped. I'll keep an eye out, but for double+ the cost of the other options I'll just deal with some noise

Looks like our requirements are matching. I only need 4 SPF+, a couple 1 GB ports and 1 or 2 PoE ports. My 7250-24P does not have the lowest idle power. But combining all my needs in one device and eliminating external PoE injectors reduces all power consumption and heat to this one device. 7150 devices I could not find for low costs but the 7250. So, I gave it a trial.

Noise still was a remaining issue. I exchanged the fans and added an additional fan to the CPU. All OK for now in the cold wintertime. But I have doubts for the summer time when temperatures are rising. Therefore, I continue to keep an eye on the fanless 7150.

 

[NickM]

Looks like our requirements are matching. I only need 4 SPF+, a couple 1 GB ports and 1 or 2 PoE ports. My 7250-24P does not have the lowest idle power. But combining all my needs in one device and eliminating external PoE injectors reduces all power consumption and heat to this one device. 7150 devices I could not find for low costs but the 7250. So, I gave it a trial.

Noise still was a remaining issue. I exchanged the fans and added an additional fan to the CPU. All OK for now in the cold wintertime. But I have doubts for the summer time when temperatures are rising. Therefore, I continue to keep an eye on the fanless 7150.

Actually..... I found a couple badly labeled ebay listings that I'm 98% confident are 7150 24-48's for $200-250. I bought one, I'll PM you the other; though I think the remaining one is non-PoE. But the non-PoE are the only fanless ones; the PoE models even with 0 PoE load are still +25W, so likely need a fan spinning

 

[D4M4EVER]

I'm working on getting VLANs setup/configrued on my IXC6610, I have updated the switch and got all the ports unlocked following fohdeesha's guides.

The current firmware version I got from the guides when updating it is ver 08.0.30tT7f3


I stubbled onto this vid by Terry Henry, he showed a new feature on how to do both Tagged & Untagged ports with out using the "Dual-Mode" option. But it looks like in order to use the Tagged & Untagged ports option in the vid iI need to be on firmware ver 8.0.80 or greater.

Can anyone confirm that the tagged/untagged on same ports will work on the IXC 6610 with out the "dual-mode" setting?
I have tried to setup my switch using it but I can't seem to get it working, it maybe just me not knowing wtf I'm doing.


I'm trying to add a total of 9 VLANs that my pfsense firewall controls the DHCP & DNS on, I want keep these VLANs controlled of pfsense for the better firewall rules managent I have setup and know how to use better then doing all the VLANs directly off the switch using L3 setup.

 

[ArmedAviator]

there's no feature differences. A long time ago in old fastiron versions, there was a premium l3 license with a few extra features, and an advanced l3 license with all the rest of the extra l3 features. the "ICX6610-ADV-UPG-LIC-SW" would provide the top advanced license if you already had and paid fr the first premium l3 license, so it would bring you up to "advanced" feature level, just like a full "ICX6610-ADV-LIC-SW" would on its own. In the new license set I got rid of the prem and then > adv upgrade, and just put a native full ICX6610-ADV-LIC-SW license on it

Which is actually all irrelevant, just looks cleaner. Several years ago they did away with the "advanced" upgrade and merged all the extra features into the premium license. so even just a premium l3 license on its own will fully unblock everything extra

as for stacking compatibility, it'll work fine. all it cares about is equal features in the end, and with fastiron 8030 even just a premium license is equal to a premium

Thanks for the explanation. I stacked the two switches together successfully and it's working as expected.

@D4M4EVER,

The 8.0.80 and up firmware is unavailable for the ICX6xxx series. If you want mixed tagged and one untagged VLAN, you just use dual-mode.

Sounds like you need to set up pfSense as VLAN-aware and tag all the VLANs to pfSense anyway. Note this method limits your inter-VLAN throughout to whatever your ofSense box can do, so don't expect wire speed routing.

 

[nivedita]

Actually..... I found a couple badly labeled ebay listings that I'm 98% confident are 7150 24-48's for $200-250. I bought one, I'll PM you the other; though I think the remaining one is non-PoE. But the non-PoE are the only fanless ones; the PoE models even with 0 PoE load are still +25W, so likely need a fan spinning

I think the 7150's can operate fanless for POE power upto 150w.

 

[tubs-ffm]

I think the 7150's can operate fanless for POE power upto 150w.

What do you mean by "can operate fanless"? By removing the fan?

Here it is about the the 24/48 port version of 7150. Only 7150-24 and 7150-48 are fanless. All PoE versions -24P, 48P, 48PF and 48ZP are with fan. The small compact switches 7150-CxxP are with PoE and fanless. But that's not what we were looking for in regards of available ports.

 

[rocketpanda40]

Just to expand on the recent discussion regarding replacement fans for the ICX 7250 and acceptable temps, I thought I'd show some of the graphs from mine captured with SNMP / LibreNMS. The switch is a 48-port, POE model, and I'm currently using 5 ports with POE (4x Cisco APs that pull ~6-6.5W, and an ICX 6450-C12-PD that pulls ~13.5W. Another 10 or so ports active, no SFP+ ports currently in use.

When I first got the switch, one of the fans was stuck and failed. I opened the switch, oiled the fan and removed dust, and all 3 worked with a large drop in temps and noise (it no longer ramped to fan speed 2). Still trying to get it a bit quieter, I replaced all the stock fans with Sunon KDE1204PKV3, but these are honestly quite crap and hardly push any air. You can see the sawtooth temperature pattern as the ASIC temp climbed to 95, fans ramp to speed 2 until temp dropped back to 85, over and over. With these fans, speed 2 isn't loud, but the constant cycling is pretty annoying and the temp change isn't great for longevity. Before replacing the stock fans, I decided to try the 60x10mm fan that's recently been discussed and have seen pretty good results so far. ASIC temp seems to have stabilized at ~57C and exhaust ~35C.

I'm going to run it a while longer and see how it works out before deciding whether to keep it in this configuration or return the stock fans, but thought some actual data would be useful for everyone to peruse.

It's worth noting that my house is generally around 65-70F, ~20-30% humidity, and the switch is in an enclosed closet that hovers closer to 75-80F and ~20-25% humidity.

 

[rocketpanda40]

What do you mean by "can operate fanless"? By removing the fan?

Here it is about the the 24/48 port version of 7150. Only 7150-24 and 7150-48 are fanless. All PoE versions -24P, 48P, 48PF and 48ZP are with fan. The small compact switches 7150-CxxP are with PoE and fanless. But that's not what we were looking for in regards of available ports.

The 7150 compact series (8/10/12 port) and some revisions of the 24/48 port non-poe models are physically fanless. The other 7150 models, POE versions included, can have the fan turned off in software, which Ruckus calls "silent mode," but it limits the POE budget to 150W to prevent overheating. See Ruckus ICX 7150-24 (ruckussecurity.com).

The command is

chassis fanless

. See chassis fanless (commscope.com).

 

[tubs-ffm]

The other 7150 models, POE versions included, can have the fan turned off in software, which Ruckus calls "silent mode," but it limits the POE budget to 150W to prevent overheating. See Ruckus ICX 7150-24 (ruckussecurity.com).

Thank you. This is great.

My current solution is like yours. In my 7250-24P I have replaced both fans by Delta EFB0412VHD-F00 and I added a Sunon MF60101V3-1000U-A99 on top of the ASIC. The noise is perfect for the final location of the device, but still not to recommend if you want to be close next to it.

For now, the temperature is OK. I am not using a lot of PoE power. But I am not yet sure when the ambient temperature in summer will raise and the final location will have less air circulation. For this reason, I am looking with one eye to a fall back solution. The 7150-24P or 7150-24 could be one.

Currently connected are: 1, PoE device, eight 1GB ports and 4 SFP+ ports.

Fan controlled temperature:
        Rule 1/2 (MGMT THERMAL PLANE): 69.0 deg-C
        Rule 2/2 (AIR OUTLET NEAR PSU): 42.0 deg-C

Fan speed switching temperature thresholds:
        Rule 1/2 (MGMT THERMAL PLANE):
                Speed 1: NM<-----> 93       deg-C
                Speed 2:        82<----->105 deg-C (shutdown)
        Rule 2/2 (AIR OUTLET NEAR PSU):
                Speed 1: NM<-----> 58       deg-C
                Speed 2:        49<----->105 deg-C (shutdown)

 

[pypypy]

ordered 6450p on ebay from this seller: didionorf on eBay

Unit was shipped in a completely empty box without any packaging material inside... Switch took a beating for sure on its way here and the box was all crushed.

Unit was clearly untested and had a failed POE board:



edit: contacting seller and will report back if/how they remedy this.


edit2: Seller issued prompt refund. Does anyone know if the POE module is replaceable? Where can I get this part?

Edit3: Last 12 POE ports are actually working. Does anyone know how to disable these system messages "Resetting module... POE info: POE Error" I don't mind using the switch for now in it's current state but I can't configure it because these messages pop up literally every 10 seconds... Tried disabling syslog but that doesn't work. Also tried "no debug all", which had no noticeable effect.

 

[tubs-ffm]

Is there any rate limit by standard for PING / ICMP responses in Fastiron?

The gateway monitoring of OPNsense shows "poor" values for the connection to my ICX 7250 running 0.80.92e. Looking on LibreNMS I also see peeks in ping time over the day. ICMP rate limit? Something wrong? Or normal due to measurement principle?

 

[Jason Antes]

Is there any rate limit by standard for PING / ICMP responses in Fastiron?

The gateway monitoring of OPNsense shows "poor" values for the connection to my ICX 7250 running 0.80.92e. Looking on LibreNMS I also see peeks in ping time over the day. ICMP rate limit? Something wrong? Or normal due to measurement principle?
View attachment 17736

I see the same issue with OPNSense and my 6610. Every 15-20th ping against any other system goes to 200ms from 1-2ms. It's kind of strange and sometimes causes an issue with Putty sessions. I figured it was an OPNSense issue with the 10Gbe card or something. Wish I knew more about troubleshooting these kinds of issues.

 

[ArmedAviator]

Hmmm, I too just loked at my ICX latency in LibreNMS and see the same wild latency spikes up to about 130-160ms. I haven't had any connectivity issues as a result, though, so it's moot to me.

 

[rocketpanda40]

Mine fluctuates alot as well, but not to the same heights y'alls seem to. It doesn't add too much to the data other than to show that your spikes up to 200ms probably have more to do with your traffic patterns and network topology than the device itself.

 

[fohdeesha]

ICMP pings to the switch itself are answered by the management CPU, which prioritizes ICMP replies very low compared to other activities, so some fluctuation is normal. That combined with the fact LibreNMS runs the ping during the same time as the scheduled SNMP polling, which puts a decent amount of load on the CPU to answer. If you just open up a command prompt and ping the switch, it should stay around 1-2ms until libreNMS's scheduled polling happens

 

[tubs-ffm]

If you just open up a command prompt and ping the switch, it should stay around 1-2ms until libreNMS's scheduled polling happens

Thank you.

My concern started when I observed the Gateway monitoring on OPNsense. (OK, my fault. What you do not know, does not make you concerned about.) Significant different values to the gateway on DSL modem than to directly connected switch. On OPNsense side both same NICs are same type and both directly connected via copper without LAGG and without VLAN in between. (see below)

Ping from Windows machine directly connected via fiber shows similar behavior. (see below)

When I will find time, I will check if there is any relation to other activities. In every case there is no load on my switch. This device is oversized for it purpose on mainly is doing nothing else than creating noise, generating heat and increasing the electricity bill.


OPNsense monitor


PING to switch

 

[NickM]

What specific type of fiber should I run in my house for 10g (maybe 40g) now, and hopefully maintaining future upgradability?

I thought I had this all figured out, and then realized the 10g SFP+ ($8) and 40g QSFP ($16) optical transceivers I was looking at had incompatible fiber runs, and now I'm secondguessing myself. I'd hoped "fiber is fiber" and you just upgraded the logic on the end.

For completeness, I have ConnectX-3 dual port QSFP cards (plus QSFP/SFP adapters), and an ICX7150 enroute. I'd considered running a direct connection between two computers via 40g QSFP in addition to the 10g connections to the switch


TLDR: Are there similarly cheap transceivers that would use the same fiber run for both 10g and 40g? What about 100g down the road?

If I should forget about the future and just run what works for now, let me know and tell me what fiber and transceivers to choose there, too