Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[rjrobert]

Hi All,

I'm brand new to these devices, and enterprise-grade networking in general, but have eager to learn something new, so I figured I'd dive in. However, I think I might have just hosed myself here and need some help. Just picked up an ICX-6450-48 off eBay and was following the Fohdeesha guide, up to the point of running the "update_primary" command when I managed to kick the power cable plugged into the wall next to my desk and the unit lost power (1st dumb mistake). Then, I realized I had my ethernet cable plugged to a regular switch port instead of the management port (2nd dumb mistake, can't read instructions, apparently). Now, I can't seem to get the serial console to show up. After many reboots, and letting the serial connection remain open for a while, I sometimes see some information come across the screen, even once prompting me to press 'b' to access the bootloader, but it seems to freeze up after that. So, I'm not sure if something happed to the serial cable (which I doubt), or I effed up the bootloader and bricked the switch.

Any ideas or suggestions?

 

[kapone]

Hi All,

I'm brand new to these devices, and enterprise-grade networking in general, but have eager to learn something new, so I figured I'd dive in. However, I think I might have just hosed myself here and need some help. Just picked up an ICX-6450-48 off eBay and was following the Fohdeesha guide, up to the point of running the "update_primary" command when I managed to kick the power cable plugged into the wall next to my desk and the unit lost power (1st dumb mistake). Then, I realized I had my ethernet cable plugged to a regular switch port instead of the management port (2nd dumb mistake, can't read instructions, apparently). Now, I can't seem to get the serial console to show up. After many reboots, and letting the serial connection remain open for a while, I sometimes see some information come across the screen, even once prompting me to press 'b' to access the bootloader, but it seems to freeze up after that. So, I'm not sure if something happed to the serial cable (which I doubt), or I effed up the bootloader and bricked the switch.

Any ideas or suggestions?

Throw it in the trash and start afresh with another one?

 

[rjrobert]

Throw it in the trash and start afresh with another one?

Yeah, figured as much. Lessons learned.

 

[kpfleming]

It's probably possible to flash the bootloader using an internal serial or JTAG header, if you want to give that a try.

 

[yobigd20]

I had an issue with my icx6650 and it was in a crash loop. I downloaded the latest firmware and followed the guide as usual, but after I flashed and set the license for ICX6650-10G-LIC-POD I noticed that it says the license capacity was only 24. This router has 56 ports on the front. Did I just lose more than half my capacity? Or is there another license somewhere I am missing?

 

[MikeSpears]

Confirmed working. (can't find em anywhere else except ebay)

Dell 40G QSFP+ to 4x10G SFP+ 3M DAC Splitter Cable DAC-QSFP-4SFP-10G-3M 27GG5 | eBay

Find many great new & used options and get the best deals for Dell 40G QSFP+ to 4x10G SFP+ 3M DAC Splitter Cable DAC-QSFP-4SFP-10G-3M 27GG5 at the best online prices at eBay! Free shipping for many products!

www.ebay.com

And wow, have they gone up in price! I bought 4 of these in 2018 for $50 shipped!

Got the cable, and getting farther. I seem to have gotten 2 servers to consistently stay linked, however anything additional seems to drop after 30-45 mins. Any ideas?

 

[kapone]

Got the cable, and getting farther. I seem to have gotten 2 servers to consistently stay linked, however anything additional seems to drop after 30-45 mins. Any ideas?

That's weird. These exact cables have been working 100% with all ports connected...since 2018 in my setup. I've never had a single hiccup.

Edit: What nics are you using on the other end?

 

[MikeSpears]

That's weird. These exact cables have been working 100% with all ports connected...since 2018 in my setup. I've never had a single hiccup.

Edit: What nics are you using on the other end?

Mellanox Connect-X2 (stopped working after about 30 mins, takes standard fiber + optics no problem)
QLogic Inc. QLogic 57810 10 Gigabit Ethernet Adapter (HPE DL360p G8 Onboard) works
Mellanox Technologies ConnectX-3 Pro EN NIC; 10GigE; dual-port SFP+ (MCX312B-XCC) (works)

 

[kapone]

Mellanox Connect-X2 (stopped working after about 30 mins, takes standard fiber + optics no problem)
QLogic Inc. QLogic 57810 10 Gigabit Ethernet Adapter (HPE DL360p G8 Onboard) works
Mellanox Technologies ConnectX-3 Pro EN NIC; 10GigE; dual-port SFP+ (MCX312B-XCC) (works)

So, the 57810 and CX3 stay connected, but the CX2 drops off? Any chance you can connect another 57810 or CX3 instead of the CX2, to rule out the cable?

 

[MikeSpears]

So, the 57810 and CX3 stay connected, but the CX2 drops off? Any chance you can connect another 57810 or CX3 instead of the CX2, to rule out the cable?

I've got an HPE 530SFP+ (same card as the QLogic Inc. QLogic 57810 10 Gigabit Ethernet Adapter (HPE DL360p G8 Onboard) except a PCI-E card, will swap that in now.

 

[MikeSpears]

Card swapped, switch rebooted (just for good measure), the 3 servers I've got connected to the breakout are all online. Hoping they stay up.

 

[MikeSpears]

31 mins and it's still working! Going to check back in an hour... @kapone any idea on what could've been up with the card?

 

[itronin]

So, the 57810 and CX3 stay connected, but the CX2 drops off? Any chance you can connect another 57810 or CX3 instead of the CX2, to rule out the cable?

I'll add to this successful configs. I am using a mix of CX3 dual, and 520's: SM and Chelsio with the Cisco AOC breakouts also Cisco CUCM DACs on the front panel SFP+'s. Zero problems for 4 years now, some in LACP some in ESXI multiple nics per vswitch and some in XCP-NG all dual connected. I have used the 40Gbe CX3's with AOC and netapp QSFP cables and zero problems on the 40gbe ports in non stacking mode.

Known issues with XL710's at 40gbe on the 6610 though they worked at 10gbe using the breakouts and that is documented in thread circa november-december 2022 and earlier.

 

[narapon]

I've got an ICX6450-48P that suddenly refuses to turn on (I found it was in an off state this morning when wifi went down), i've only had it installed 1 day prior and power is from a delta UPS (other equipment attached not affected). Now when I try to turn it on, I can hear the distinct click from the PSU area, assuming that it turned on. I've tried removing potential dust by using a datavac (holding down the fan blades of course) in the hopes that it was just dust causing issues (seemed like that was the case when the console port would not work)

Where do I go with diagnosing this? I saw 2 EPS power connectors but those will require the power shelf, and I assume they are not wired according to anything standard, power supplies for this are probably hard to come by

(update with IXC6450-48P pictures later)

update: it's powered on now after pressing the reset button, have no idea why

update again: it's turned off again, could it be heat from neighbouring devices?

I've also got a ICX7250-24P that cannot detect the POE module (firmware version shows 0.00), I have taken out the POE module to inspect for any potential physical damage and it looks like its in pristine condition which leads me to think its a firmware issue? I've tried flashing various firmware versions both UFI and pre-UFI images, neither of them pick up on the POE module, when I try to flash the POE fw via inline tftp on the pre-UFI version, i get a soft lock.

http://imgur.com/a/P6fYahn

ICX7250-24P

ICX7250-24P POE module bottom

ICX7250-24P POE module top

 

[kapone]

31 mins and it's still working! Going to check back in an hour... @kapone any idea on what could've been up with the card?

Excellent. It's gotta be something janky with that card. CX2s generally play nice with almost all optics. Like I said, exact same cable, ~5 years up time...zero issues.

I've absolutely zero idea about why that card wouldn't play nice, except....it's janky?

 

[fohdeesha]

Hi All,

I'm brand new to these devices, and enterprise-grade networking in general, but have eager to learn something new, so I figured I'd dive in. However, I think I might have just hosed myself here and need some help. Just picked up an ICX-6450-48 off eBay and was following the Fohdeesha guide, up to the point of running the "update_primary" command when I managed to kick the power cable plugged into the wall next to my desk and the unit lost power (1st dumb mistake). Then, I realized I had my ethernet cable plugged to a regular switch port instead of the management port (2nd dumb mistake, can't read instructions, apparently). Now, I can't seem to get the serial console to show up. After many reboots, and letting the serial connection remain open for a while, I sometimes see some information come across the screen, even once prompting me to press 'b' to access the bootloader, but it seems to freeze up after that. So, I'm not sure if something happed to the serial cable (which I doubt), or I effed up the bootloader and bricked the switch.

Any ideas or suggestions?

getting hosed while flashing the primary OS image won't do any harm to the bootloader, which is the important bit. unplug and replug your serial adapter (the usb side), then reboot your computer, then unplug the switch from the wall completely, then plug it in and smack b. you should get a prompt eventually. Your weird serial output sounds like what happens when a lot of these cheap usb to serial adapters get jostled or bugged out

 

[fohdeesha]

I had an issue with my icx6650 and it was in a crash loop. I downloaded the latest firmware and followed the guide as usual, but after I flashed and set the license for ICX6650-10G-LIC-POD I noticed that it says the license capacity was only 24. This router has 56 ports on the front. Did I just lose more than half my capacity? Or is there another license somewhere I am missing?

The ICX6650 comes with 32 ports enabled by default, the biggest PoD license possible is a quantity of 24 which unlocks the remaining 24, for a total of 56. fastiron-08030-licensingguide.pdf in the same zip you downloaded

 

[MikeSpears]

Excellent. It's gotta be something janky with that card. CX2s generally play nice with almost all optics. Like I said, exact same cable, ~5 years up time...zero issues.

I've absolutely zero idea about why that card wouldn't play nice, except....it's janky?

Who knows, who cares. I threw it in the trash lol

 

[NateR]

Hello all,

I've got an ICX6450-24p I've had in use for a year or so now. I've noticed that when I mess around with stuff in my rack, sometimes I'll lose the SFP ports (no status lights), which I use for uplink to the upstream switch, requiring a reboot of the switch. I believe I've finally figured out what is causing it - I disconnected a shielded network cable the other day, and happened to touch the metal shielding on the connector and shocked it with ESD (it's pretty dry here, and I typically touch the rack before anything else to discharge it, but either forgot to or just moving around built up a charge). The SFP ports went dark when this happened. Rebooted, and they came back up.

Any ideas on how to keep this from repeatedly happening?

Also, has anyone found a good alternative to OEM rack-mount ears? Mine came with a couple, but as I didn't have a rack at the time, I stupidly removed them and put them "where I will remember them..." well, after much searching, I have failed to find them.

Thanks

Anyone have any ideas about either of these?

Thank you

 

[rigel.local]

Please help to understand PSU fan airflow direction. I found 2 variants on ebay with PSU airflow green arrow down with letter E and orange arrow up with letter I. Does E stands for exhaust and I for intake? Does it affect noise level and what it the less loud version?