Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[Marc_]

Morning all. Have a strange issue with a 6450-48P. Powering a few things via PoE but I'm having random disconnects.

PoE: Power disabled on port 2/1/38 because of PD disconnection.

PoE: Power enabled on port 2/1/38.

PoE: Power disabled on port 2/1/38 because of PD disconnection.

PoE: Power disabled on port 2/1/40 because of PD disconnection.

PoE: Power enabled on port 2/1/38.

PoE: Power enabled on port 2/1/40.

Anyone seen this before?

 

[MikeSpears]

If anyone needs one of the Dell 40G QSFP+ to 4x10G SFP+ 3M DAC Splitter Cable DAC-QSFP-4SFP-10G-3M 27GG5 cables let me know, I've got one I don't need and am willing to let it go for $50 shipped anywhere in the US. Ended up swapping it with some shorter ones purchased off Amazon.

 

[gr2659]

So... I'm struggling trying to use my ICX6610s with some 40G Cisco NICs. Hoping someone has gone through this and can help?

I have (2) ICX6610s on 08.0.30uT7f3. They were stacked. I just broke the stack as part of my troubleshooting.

I'm trying get a 40G link up between ports 1/2/1 or 1/2/6 of my ICX6610s and my Cisco 40G NICs (I have both an mlom and vic version of the 40G NICs.) on a couple of C240-M4 servers. I have tried both NICs on (2) servers and cannot get a link to come up. The server shows the NICs and shows that it recognizes the optics and they show as "supported". They being (3) different QSFP setups... I have a Brocade branded 40G QSFP, a Cisco branded 40G QSFP and a handful of the KAIAM QSFPs connected with fiber. I've used both DACs to stack my 6610s using those same ports so I know the ports and DACs work.

Not being able to get these to come up on either switch, I pulled the 2nd out of the stack, reset the config on one to factory settings and removed the stacking configuration. The results are the same...

telnet@brocade48-1#sh int eth 1/2/6
40GigabitEthernet1/2/6 is down, line protocol is down
  Port down for 8 minute(s) 49 second(s)
  Hardware is 40GigabitEthernet, address is 748e.f893.efe6 (bia 748e.f893.efe6)
  Interface type is 40Gig Fiber
  Configured speed 40Gbit, actual unknown, configured duplex fdx, actual unknown
  Configured mdi mode AUTO, actual unknown
  Member of 3 L2 VLANs, port is tagged, port state is BLOCKING
  BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
  Link Error Dampening is Disabled
  STP configured to ON, priority is level0, mac-learning is enabled
  Openflow is Disabled, Openflow Hybrid mode is Disabled,  Flow Control is enabled
  Mirror disabled, Monitor disabled
  Mac-notification is disabled
  Not member of any active trunks
  Not member of any configured trunks
  No port name
  MTU 1500 bytes, encapsulation ethernet
  300 second input rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
  300 second output rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
  0 packets input, 0 bytes, 0 no buffer
  Received 0 broadcasts, 0 multicasts, 0 unicasts
  0 input errors, 0 CRC, 0 frame, 0 ignored
  0 runts, 0 giants
  0 packets output, 0 bytes, 0 underruns                         
  Transmitted 0 broadcasts, 0 multicasts, 0 unicasts
  0 output errors, 0 collisions
  Relay Agent Information option: Disabled

Anyone have any thoughts on what I might be missing?

 

[thegardentool]

Could anybody direct me to some good resources to guide me on best practices of using an L3 switch along with a firewall appliance? My Asus N router is finally starting to give me some trouble and for the short term my plan is to use the J6413 box I ordered earlier this week to branch out into something a bit more advanced with OPNsense while using the router in AP mode. Longer term I've been debating on just taking the plunge on a C12P along with a Ruckus AP so I could also expand out to some security cameras later. I had looked at the switch and AP a couple years ago it just wasn't priority and was hopeful that UBNT would have a bit more friendly SOHO non rack solutions come out as I liked the idea of having network and video integrated into one place, but alas the Dream Router and Dream Wall just weren't the solutions for me.

 

[juju]

HAProxy or something like that (I use nginx). Do you announce static IP range through your ISP? Because if you only have 1 IP and serve multiple websites I imagine you'd use some kind of front end proxy to route traffic?

I am not asking about how to route external traffic to the internal webserver. I am asking if it makes sense to isolate that webserver into its own vlan ( and all other web facing servers ), and use acls to block traffic to other internal parts. I wanted advice on how to structure that acl. I am not sure how to setup that acl.

 

[CHANABRA]

Hello all just dropping a line, thank you all for all the posts and information I can absorb... I appreciate it!!
Also, I just sent a message requesting license keys

Grabbed a ICX6610-24E for $150 fully functional....seemed like a nobrainer compared to the CRS328-24P when I need full L3....and then some.
hoping I can figure a way between replacing thermal paste and optimizing cooling to get this thing a bit more silent

 

[NablaSquaredG]

Also, I just sent a message requesting license keys

Why would you do that?

It's literally the first thing in this thread to NOT send PNs and instead use this thread.

 

[hmw]

Could anybody direct me to some good resources to guide me on best practices of using an L3 switch along with a firewall appliance? My Asus N router is finally starting to give me some trouble and for the short term my plan is to use the J6413 box I ordered earlier this week to branch out into something a bit more advanced with OPNsense while using the router in AP mode. Longer term I've been debating on just taking the plunge on a C12P along with a Ruckus AP so I could also expand out to some security cameras later. I had looked at the switch and AP a couple years ago it just wasn't priority and was hopeful that UBNT would have a bit more friendly SOHO non rack solutions come out as I liked the idea of having network and video integrated into one place, but alas the Dream Router and Dream Wall just weren't the solutions for me.

I'll do a write up later but I managed to get OpnSense working with a US-XG-24 and a Brocade ICX-6610 for my homelab. Don't know if that is what you're looking for.

At some point when the prices for ICX7650s have dropped to reasonable levels, I will just go with that. It's a headache getting anything Ubiquiti working properly. Except for their Unifi Video, I intend to ditch Ubiquiti and move away from their APs and switches when economically feasible

 

[azmaveth]

Talking to a vendor on eBay about a ICX6610 they are selling and they said that they get this when booting:

Monitordir
1740 [fccb] $$sshhost.key
364 [d646] stacking.boot
8657 [8af8] startup-config
8648 [a2e1] startup-config.backup
2658 [01e4] startup-config.legacy
22067 bytes 5 File(s)
65142784 bytes free
Monitorboot system flash primary
`File not found, 'primary'
Monitorboot system flash secondary
`File not found, 'secondary'

Am I right in thinking this is basically bricked since both flash memories are wiped/corrupted? Is there a way for me to restore the firmware without hardware tools?

 

[NablaSquaredG]

s there a way for me to restore the firmware without hardware tools?

My guess would be that the u-boot bootloader is still there and working, so you should be able to flash the primary / secondary image from the bootloader (as described in @fohdeesha's guides)

But that's just a guess... Could be that the switch is broken.

Possibly related:

https://forums.servethehome.com/index.php?threads/issues-with-an-icx6610-dont-boot.21653/

 

[i386]

The switches from the op are now about 10 years old (The oldest pdf I found was dated 2013 from brocade). I'm wondering how long the components used for these icx switches will last...

 

[zunder1990]

The switches from the op are now about 10 years old (The oldest pdf I found was dated 2013 from brocade). I'm wondering how long the components used for these icx switches will last...

Given that the end of sale was 2018 I am betting they was still making new 6610 in 2018, alot of those switches still floating around may not be that old.

 

[thegardentool]

I'll do a write up later but I managed to get OpnSense working with a US-XG-24 and a Brocade ICX-6610 for my homelab. Don't know if that is what you're looking for.

At some point when the prices for ICX7650s have dropped to reasonable levels, I will just go with that. It's a headache getting anything Ubiquiti working properly. Except for their Unifi Video, I intend to ditch Ubiquiti and move away from their APs and switches when economically feasible

Well I do think I’ll likely skip UBNT stuff now completely. I hadn’t invested in any of it while waiting to see if anything new would have worked better. I suppose I’ll get it all figured out. May just end up still picking up one of these C12P switches and start off running it just as an L2 while learning OPNsense and then figuring out down the line if it would be better to move the routing back to the switch.

 

[msg7086]

Sunon PSD1204PHB1-A(2).Z.F.PWM.GN "Tiny Terror" - also currently unobtainable
These are new design MagLev parts, and really impressive. 40x40x15mm (so the thinnest here,) but 14CFM, 0.63in H2O, and 44.2dBA @ 1m at 12,000RPM. They're also a LOT easier than the Mechatronics; Sunon MOQ is just 30 for a non-custom part, expect around $10-12/ea. These are awesome fans if you can get your hands on them and can stand the noise.

Hey, I just got some of these but they seem to have 4 leads (red, yellow, black, blue). Any suggestion on how to wire them to ICX6450-48P?

Thanks!

 

[NablaSquaredG]

I'm just reading through the guides, but I want to make sure I miss nothing

What security features are supported by FastIron that can be used in public networks (aka everybody can connect to the switch gigabit ports)?

Stuff like
BPDU Guard
Root Guard
Port Security (MAC Address Limit)
DHCP Snooping
Dynamic ARP Inspection
IP Source Guard
Segmentation into multiple VLANs


I'm currently trying to figure out what is the best way to secure access to management (no out of band management possible)

 

[kpfleming]

The docs for the FastIron software are publicly available, so you can review them to determine whether the features you desire are available (some may have different names than what you used). Also keep in mind that the software has different features for different families of devices: for example the L3 software for 71xx/72xx devices doesn't support BGP but the 73xx and above do.

VLANs are definitely supported. If you don't have any OOB management, you'll be stuck with SSH and will have to secure it using normal SSH mechanisms. The SSH implementation in the 08.x series firmware is a bit dated and doesn't support modern SSH features like elliptic curve keys, but it's otherwise functional.

 

[NablaSquaredG]

The docs for the FastIron software are publicly available, so you can review them to determine whether the features you desire are available

Guess what I've been doing (And if you look carefully, you'll find all of them in the FastIron manuals)

 

[PANiCnz]

My ICX 6450 is shutting down once a week. I have to disconnect it from the power and leave it for several hours before it will turn on again. Assumed it might be heat related, but the last time it shutdown was a cool day, and the room was well ventilated. Has happend three times now and seems to be happening about a week apart.

Any ideas/tips on how to try and investigate the problem?

 

[i386]

Did you Check for dust in the vents/openings?

 

[PANiCnz]

Did you Check for dust in the vents/openings?

Yip pulled it apart and everything was reasonably clean, definitely not blocked.