Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[kpfleming]

When I tried using 1gbps copper SFPs, I was unable to get the switch to stack; it complained the link wasn't 10gbps. I was hoping someone here might have experience, or eldritch knowledge, about whether I could use a cheap 10gbps SFP that might report 10gbps despite only being connected at 2.5gbps, and rate limit those interfaces to prevent drops. Anybody been down this road? If so, which SFPs do you like for this kind of adventure?

You may be out of luck there. IIRC the switch stacking links don't even run in normal 'Ethernet' mode, they use something called HiGig and it isn't a protocol that you will be able to pass across bridges.

 

[Shaun Bosworth]

Hi Guys

Okay the mystery deepens, I can see that my Windows DHCP server is issuing IP addresses to the Intel 40G link cables, but the card within the OS does not acknowledge that the leases has been issued, as it only shows an AutoIP.


I have confirmed that the switch is linking at 40G to the card on both ports

Current configuration:
!
ver 08.0.30uT7f3
!
stack unit 1
module 1 icx6610-48p-poe-port-management-module
module 2 icx6610-qsfp-10-port-160g-module
module 3 icx6610-8-port-10g-dual-mode-module
stack disable
!
global-stp
!
!
!
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
spanning-tree 802-1w
!
!
!
!
!
aaa authentication web-server default local
aaa authentication enable default local
aaa authentication login default local
enable aaa console
fast port-span exclude ethe 1/2/1 ethe 1/2/6
hostname brocade-sw
ip dhcp-client disable
ip dns domain-list bozzy.info
ip dns server-address 192.168.10.15 192.168.10.16
ip route 0.0.0.0/0 192.168.10.1
!
no telnet server
username admin password .....
!
!
clock timezone gmt GMT+02
!
!
ntp
disable serve
server 192.168.10.1
!
!
web-management https
web-management refresh port-statistic 10
web-management refresh stp 10
web-management refresh tftp 10
web-management refresh rmon 10
web-management frame bottom
web-management page-size 20
!
!
router rip
!
!
!
!
!
!
!
interface ethernet 1/2/1
no spanning-tree
no flow-control
!
interface ethernet 1/2/6
no spanning-tree
no flow-control
!
interface ethernet 1/3/1
speed-duplex 10G-full
!
interface ethernet 1/3/2
speed-duplex 10G-full
!
interface ethernet 1/3/3
speed-duplex 10G-full
!
interface ethernet 1/3/4
speed-duplex 10G-full
!
interface ethernet 1/3/5
speed-duplex 10G-full
!
interface ethernet 1/3/6
speed-duplex 10G-full
!
interface ethernet 1/3/7
speed-duplex 10G-full
!
interface ethernet 1/3/8
speed-duplex 10G-full
!
interface ve 1
ip address 192.168.10.7 255.255.255.0
!
!
!
!
!
!
!
!
!
end

This is the current switch config. I know a lot of you will say that I should have gotten a Mellanox card, but I prefer to use Intel NICs. Also, just a note that this XL710 is a Dell OEM model, obviously used in a Dell PowerEdge server, along with Dell 3M QSFP DAC cables, so i think there shouldnt be any compatibility issues.

Any thoughts or resolution would be appreciated

 

[kevindd992002]

I see that the ICX6610 has 4 different versions:



Questions:

1) What is the use case of an "intake air flow"? I though all rack switches have their rear fans exhaust airflow out?
2) What's the difference between Base and Premium software? Does it matter for home lab use?

 

[i386]

1) What is the use case of an "intake air flow"? I though all rack switches have their rear fans exhaust airflow out?

It's for managing cooling in a datacenter: all network ports are on the same side of the rack (servers AND switches) AND the air flows in the same direction

 

[kevindd992002]

Ok, so why would the fan be intake if all network ports are on the same side of the rack? I guess I still don't understand the use case. And what I want is the standard exhaust airflow switch if I have the switch ports in front and the server ports at the back, correct?

 

[i386]

Ok, so why would the fan be intake if all network ports are on the same side of the rack?

If you have a switch with front to back airflow and use it as a tor switch it would pull hot air into a server rack -> more heat -> higher risk for failure (not just the switch but all other stuff inside the rack)

And what I want is the standard exhaust airflow switch if I have the switch ports in front and the server ports at the back, correct?

Yes.
But if you have many devices connected the cable management will be a lot harder and (you would need longer cables going from front to back of the rack)

 

[klui]

In a lot of cases switches are mounted in the rear of a rack in a data center. It helps with minimizing cable length and if they are mounted that way, front-to-rear airflow will route server exhaust air through the switch. That's why there are rear-to-front airflow SKUs to go along with vented blank panels so air from the cold aisle can properly cool a switch mounted in that configuration. And while the term is "top" of rack, they may not be mounted there, instead in the middle for the same reason re: cable length.

 

[kevindd992002]

Ok, I think I have a good idea of what you guys are saying. I have just a couple of devices in my home rack. A Synology NAS, NAS extension bay, and pfsense. I think I'm set with the ICX6610-48p-E (exhaust airflow) then so all my devices have front-to-back airflow.

 

[fohdeesha]

airflow in an air conditioned house with like one other device in the rack doesn't matter air isn't even real lmao. have you ever seen it? think not pal

 

[juju]

I have an IoT device on my network which is not using the dns server on the ICX7250 and always defaults to 0.0.0.0. ( so cant resolve dns names). All other devices on my network correctly use the dns server ip I have setup on my 7250. Other devices on the same vlan as this IoT device have no such issue with dns resolution. Is there a way to force a specific device to use the dns server of its vlan ?

 

[Rttg]

I have an IoT device on my network which is not using the dns server on the ICX7250 and always defaults to 0.0.0.0. ( so cant resolve dns names).

Are you sure the IoT device isn’t trying to use a hardcoded DNS server?

Some will to ensure they can ‘phone home’ (and avoid ad blockers). If that’s the case, you may need to use a DNAT rule to rewrite DNS traffic to your preferred server(s).

 

[juju]

Are you sure the IoT device isn’t trying to use a hardcoded DNS server?

It is entirely possible it is - I suspect that is the issue. In this case, it has a default dns server of 0.0.0.0 ( if you query the device for dns server info). Not sure if its returning 0.0.0.0 because its not getting the dns info from the ICX switch or its hardcoded. Can you give some pointers how to do the DNAT rule ? Where ? On my pfsense switch or on the ICX7250?

 

[Rttg]

Can you give some pointers how to do the DNAT rule ? Where ? On my pfsense switch or on the ICX7250?

The ICX can’t do NAT - it’d have to run on your firewall/internet gateway.

Can’t say how to do it on pfsense - I’m running VyOS so don’t know the syntax/config there.

 

[seatrope]

Hi all, thanks for the accumulated wealth of information. Have had a ICX6610 and a couple of ICX6450s for a while now but just started to use the L3 features.

I've gotten VLANs and inter-VLAN routing on the ICX6610 up and running, with pfSense only serving firewall duties. DHCP/DNS is via a piHole with dnsmasq serving all the subnets.

I have been searching and can't figure out how to do this. @fohdeesha any help would be appreciated - thanks again for your help with the switches a few years back!!
I either need to:
1) Stack the ICX6610 and one ICX6450 (i know only the front ports can be used as a "peri-trunk") but still preserve all the back 40Gbe ports for data use (I have a couple of servers connected via 40Gbe 1/2/1 and 1/2/6). It won't let me enable stacking without pulling the back ports into it too. I saw where you have used one 40Gbe and one breakout port for stacking btw two ICX6610 but this is somewhat different, i guess.
OR
2) if the above is impossible, what's the best way to pass all the VLANs defined on the 6610 to the 6450 and distribute them as untagged ports? I tried making the link between the 6610-6450 a dual-mode port on both sides with the same VLAN numbers - that did not work and I could not ping the 6610 interface IP after I did that.

Apologize in advance for the n00b questions but am slowly learning L3 networking.

Thanks so much!

 

[seatrope]

Well, I figured out 2) myself. Stupid mistake of course - should just have tagged all the vlans and not used dual-mode for the "uplink".
Also, forgot to assign IPs on the vifs.

But still have no idea how to do 1). Would be nice to connect 2 10G ports in a "ring" topology stack btw 6610 - 6450.

Thanks!

 

[sfrode]

I blew the power supply on my ICX7250-48 a while back. Don't know why it happened, but It might have been temp related since the switch was pretty toasty when it happened.

In case anyone else ends up in the same situation; FSP200-P35-A12 can be used as a replacement with a couple of mods:

• need to swap 12V/GND on the internal power plug between PSU and EPS board
• need to extend the internal power cable between the PSU and EPS board or just move the EPS board

 

[kfriis]

I have noticed that "sometimes" after a complete power outage, the ICX-6450 will not restart/reboot. The only way to turn the switch ON again is to unplug the power cord and plug it right back in. After this, the switch boots and functions normally. But it will not "take power" or reboot without pulling the power cord.

Has anybody experienced this? Is there some kind of configuration I should be checking to ensure that the switch will reboot/restart after a power outage?

Thank you!

 

[heromode]

I have noticed that "sometimes" after a complete power outage, the ICX-6450 will not restart/reboot. The only way to turn the switch ON again is to unplug the power cord and plug it right back in. After this, the switch boots and functions normally. But it will not "take power" or reboot without pulling the power cord.

Has anybody experienced this? Is there some kind of configuration I should be checking to ensure that the switch will reboot/restart after a power outage?

Thank you!

only thing i can think of is it's trying to power on at the exact same time as all your other gear. I remember @fohdeesha mentioning that these things are designed for 'pulling the plug' power recycling, and no need to be afraid of that. Maybe the PSU just doesn't get enough juice the exact second all your other gear powers on?

 

[fohdeesha]

I have noticed that "sometimes" after a complete power outage, the ICX-6450 will not restart/reboot. The only way to turn the switch ON again is to unplug the power cord and plug it right back in. After this, the switch boots and functions normally. But it will not "take power" or reboot without pulling the power cord.

Has anybody experienced this? Is there some kind of configuration I should be checking to ensure that the switch will reboot/restart after a power outage?

Thank you!

there's no "power-on" for these things, if power is applied to the board, they boot. Are these really brief power flickers, like a second or so? All I can think of is it's losing power just shortly enough that the voltage supplied to the CPU board doesn't drop enough to completely (the PSU is going to have ~400ms of hold-up time) to reset the CPU and it gets stuck in a half-reset state. have ran into this on a lot of electronics with very brief power flickers that don't quite drop voltage completely to micros/CPUs

It's also possible when your power is returning it's high voltage (or sagging) to the point the PSU is going into protection, and the PSU is not going to output power until it's reset (power removed and applied again)

 

[jbrukardt]

e

So in order...
- the best fans for this are unobtanium due to COVID-19, but don't use Noctua. Noctua is overpriced junk - it has the lowest flow rate and lowest static pressure. Anything from Sunon or NMB or Nidec or Delta is fine. They also cost far less.
- 08.0.30t for the 6450 is the maximum possible
- see first post for that, but if the switch was only configuration reset and not factory reset it may have them installed. Do a show licenses first always.

Edit: to expound upon my 'unobtanium' fan choice, bear in mind, I have been doing systems integration for 30 years and have VERY extensive rackmount equipment experience. I know why Brocade chose the fans they did, I have worked with those fans, and they are the same fans I would have chosen. (BTW, at speed 2 the 6450-40P is attempting to move 90.9CFM. Yes, each of those 40mm fans is rated for 30.3CFM free air.)
If you want to quiet down these units, the BIGGEST change you can make is to remove the stamped grill with a Dremel. It's all sharp edges, and significantly blocks the hub as well. Turbulence from things like grills are what generate serious noise. If you still need protection, tack on a wire grill on the OUTSIDE of the chassis. HOWEVER, this will result in slightly reduced cooling to the front right corner. I won't bother going into my unobtanium choice, because unless you're ready to order at least 250 of them, you can't buy them.

So here's some fans you can buy that I'd recommend.

• Delta EFB0412VHD-F00 - $12.40/ea @ Digikey
  There's a REASON Delta's a top pick for ODM and OEM. 40x40x20mm, 10.1CFM, 0.416in H2O (which is insane,) 32.5dBA @ 1m, rated for 70k hours at 50C. Make sure it's THAT part number and not the Rev C, which is a vastly inferior part.
• Comair Rotron "Gryphon" GDA4028-12BB - discontinued, alas
  If you can find these? BUY THESE. 40x40x28mm, but 11CFM, 0.34in H2O, 8800RPM, but just 31.4dBA! I am still mad they discontinued them. No, I am not selling any of my spares.
• Mechatronics MR4020E12B1-RSR - $7.50 @ Digikey (when in stock)
  READ THE PAGE. These are non-stocked currently! However these offer excellent balance between flow and noise. 40x40x20mm, 15.8CFM, 0.45in H2O, and 39dBA @ 1m at 11000RPM. (Yes, it's about the quietest 11k fan ever made.) You need the E12B1 though - NOT the B2. The B1 has a tach, the B2 is a rotor lock wire.
• Mechatronics MR4020H12B1-RSR - about $8
  The 'slightly slower' version pushing 13.6CFM at just 35.1dBA. Note that you cannot use the B1+6 that DigiKey stocks. The B1+6 is a 4-wire PWM, and the ICX's 12V control will burn up the motor.
• Mechatronics MH4028L12B1-RSR - PLEASE TELL US ALL IF YOU FIND A SOURCE!
  These. Are. GLORIOUS. 40x40x28mm, 12.81CFM, 0.37in H2O, 39dBA, I just LOVE these fans. But nobody stocks the low speed version. And the next step up (the M) is already over 45dBA. But they have amazing harmonics due to the 4028L being less than half the speed the frame was designed for (Max 16,500RPM!)
• NMB-MAT 1608VL-04W-B69-B00 - $10.47 @ Digikey
  These are an excellent 'middle of the middle' choice. 40x40x20mm, 11.3CFM, 0.399in H2O, 34.6dBA @ 1m at 9500RPM, and 40k hour lifetime. They're also generally stocked by multiple vendors. Really great fans for general use, but believe me, that 40k hour lifetime is near spot on - expect to replace every 3 years or so.
• Sunon PSD1204PHB1-A(2).Z.F.PWM.GN "Tiny Terror" - also currently unobtainable
  These are new design MagLev parts, and really impressive. 40x40x15mm (so the thinnest here,) but 14CFM, 0.63in H2O, and 44.2dBA @ 1m at 12,000RPM. They're also a LOT easier than the Mechatronics; Sunon MOQ is just 30 for a non-custom part, expect around $10-12/ea. These are awesome fans if you can get your hands on them and can stand the noise.

And while we're here, let's talk about the Sunon KDE MagLevs. Which will be a very short talk: KDE MagLevs are terrible for this application. They're cheap, plentiful, and useless. They are NOT designed for applications like this. The KDE1204PKVX that you can find anywhere and everywhere? It is objectively the worst. 10.8CFM, 27.5dBA, sounds great, right? It also has a static pressure of 0.27in H2O. These were designed for very free flow applications. The ICX chassis grills are the opposite.
I also don't recommend SanAce because the PSUs you all complain about on the 6610's? Those are SanAce fans. SanAces have a very distinct harmonic that everyone hates. That harmonic is also why SanAces perform the way they do. You can't have one without the other. If you think those 6610's are bad, at least there is much, much worse.

As far as selecting fans? The closer to stock flow, the better, however you achieve it. Installing 10CFM peak fans in an ICX6450-48 or 48P is a net cooling reduction of over 70%! So, you know, don't do that unless you want to run very hot or are not using POE! This is why I make angry noises at people who think they can just swap fans around in equipment like this. For the love of god people, the ICX6610 fan modules? 2x Delta FFB0412UHN's per module, that's over 50CFM per module, over 100CFM excluding the power supplies! Nobody puts that kind of brute force in unless they NEED it. Especially as those FFBs are close to $20/each at quantity.
But as I said: how you get there doesn't matter as long as you get there or accept the trade-off. Ducting a pair of 45CFM 120mm fans on the side of your 6450-48P? That'd work too. No, seriously! (Won't work on the exhaust side because you just don't have enough opening and it's too much pressure loss.) Add more, quieter fans? Also valid. Strap tiny quiet fans to the heatsinks? Absolutely will help. Only care about performance at all costs? Throw in some contra-rotating fans! Want to get stock cooling performance out of a 6450-48? Cut out 2 more fan holes, drop in 3 x 9CFM, dead silent and ready to rock.

FYSA, the Comair's are available at

12V .20A 2.4W Gryphon Series DC Fan w/Tachometer Comair Rotron GDA4028-12BB

my order just shipped.