Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[ekv]

I'm not a networking guy. Two months ago (on a whim) I picked up a MikroTik CRS326 and ConnectX-2 cards, and the SFP+ just works. Shocked, I immediately started looking for more and affordable SFP+ ports. Hence looked at LB6M though bought Brocade icx6450-12p (hoping for lower power consumption).

I go to "Fohdeesha Docs" -- which is mind-bogglingly, take-my-money-please amazing -- and one of the first things is "Connect to the switches serial/console port..."

Umm, ok.

This may be a (very) dumb question, but which is the serial port? I've looked through the 6450 docs and ... there appears to be no labeling, just icons. I'm guessing the top (left-most) port is the serial, and the one just below it is the mgt. port.

The next question then is, since my computer(s) has no serial port, how do I connect?

There's this
USB to RS232 Serial to RJ45 CAT5 Console Adapter Cable for Cisco Routers FTDI | eBay
But is that actually converting anything? [I literally have no COM ports in my (w10) Device Mgr.] So would I instead need a combo RJ45-to-DB9 cable and then a DB9-to-USB cable? Like this

TRENDnet USB to Serial Converter, USB 1.1 to RS-232 Male (9-pin) DB9 Serial | eBay

9-Pin DB9 Serial RS232 Port to RJ45 Cat5 Ethernet LAN Rollover Console 797734232417 | eBay

Thanks in advance. [Already installed PuTTy and Tftp via Chocolatey]

 

[KFJ]

I recently purchased a used ICX6610-48P-PE with the PREM and 10G licences. If I update the firmware to latest, will the licences still be valid/unaffected? Do I need to back up the licences prior to updating the firmware?

 

[arglebargle]

Does anyone know how to trap chassis notifications? The Fan Up/Down notifications are making it hard to concentrate on configuring my switches.

This is what I'm talking about:

Rear Side Fan 1  is down
Rear Side Fan 1  is up
Rear Side Fan 1  is down
Rear Side Fan 1  is up
Rear Side Fan 1  is down
Rear Side Fan 1  is up
Rear Side Fan 1  is down
Rear Side Fan 1  is up
Rear Side Fan 1  is down
Rear Side Fan 1  is up
Rear Side Fan 1  is down
Rear Side Fan 1  is up
Rear Side Fan 1  is down
Rear Side Fan 1  is up
Rear Side Fan 1  is down
Rear Side Fan 1  is up
Rear Side Fan 1  is down
Rear Side Fan 1  is up
Rear Side Fan 1  is down
Rear Side Fan 1  is up

core02(config)#
core02(config)#Rear Side Fan 1  is down
Rear Side Fan 1  is up
Rear Side Fan 1  is down
Rear Side Fan 1  is up
Rear Side Fan 1  is down
Rear Side Fan 1  is up

what switch model is that? I've never seen fan up/down messages on any brocade model unless I physically pulled out a fan tray during operation, does one of your fan trays have a loose connection or something?

So I investigated this a bit more today and as far as I can tell there's no issue with the fan actually having a loose connection, the firmware just perceives that the fan is down. I ran the switch with the cover off on my desk for an hour and the fan was definitely spinning while the firmware declared that it was failed.

The only thing I can think of here is that my silent fan is running at too low an RPM for the switch to perceive it as working. I guess I can test with the slightly higher RPM fan that I bought to use with my -24P but never ended up installing.

Is there a way to stop the switch from sending logs to the console connection but keep the notices in the log? That'd at least let me work on the CLI without these interruptions. For the moment I'm just using "no logging buffered debug" and "no logging buffered informational" ... it just seems like a dumb idea to add "no logging buffered alerts" in case something actually breaks.

 

[fohdeesha]

So I investigated this a bit more today and as far as I can tell there's no issue with the fan actually having a loose connection, the firmware just perceives that the fan is down. I ran the switch with the cover off on my desk for an hour and the fan was definitely spinning while the firmware declared that it was failed.

The only thing I can think of here is that my silent fan is running at too low an RPM for the switch to perceive it as working. I guess I can test with the slightly higher RPM fan that I bought to use with my -24P but never ended up installing.

Is there a way to stop the switch from sending logs to the console connection but keep the notices in the log? That'd at least let me work on the CLI without these interruptions. For the moment I'm just using "no logging buffered debug" and "no logging buffered informational" ... it just seems like a dumb idea to add "no logging buffered alerts" in case something actually breaks.

are they getting sent to the physical serial port or the telnet/ssh session? Either way probably no way to silence them, on purpose, I can't think why you would ever want to not want to know about a critical piece of the switch failing (other than well, because you replaced the fans and it's wrong )

you could just cut the RPM sense wire, that way it'll think it has failed right off the bat on boot and never report again. Or get a faster fan

 

[fohdeesha]

ICX6650​

• $300 (or even less occasionally)
• 64x 10gbE (56 via front SFP+, 8 via 2x rear QSFP breakout)
• 4x 40gbE via rear QSFP
• 120w power draw
• a little louder than the icx6610
• all the advanced software features of the icx6610 (vrf's, bgp, tunnels, etc)
• takes same PSU and fan tray models as the ICX6610 (same rack ears too)
• does NOT support stacking
• DOES support MC-LAG (LACP across switches) (Brocades term is MCT)

a little much for most home users, but a good deal if you have a lot of 10gb to aggregate

this seller accepted a BO of $250. It was listed and shown as no rack ears, but came with some. Can't promise his remaining stock will as well - Brocade ICX6650 56-Port 10/40 Gigabit Ethernet Top-of-Rack Switch 42111395699 | eBay


soul extraction:

 

[Matt G]

I'm not sure what I'm getting myself into but this thread was just so great I've ordered three icx6610 from two different sellers... all the way from the states to the north ass-end of Sweden. I guess I should start reading the "fine print" on page one before I get burnt to a crisp for stupid questions. From what I understand the last two I ordered should have atleast one 4-port license upgrade for the sfp ports.

I sort of got into to this because I got tired of my synology-devices being crap. I bought two Dell r720xd, followed by some reading just to realize I can't keep doing 1gbe anymore because... upgrades! So now I'm stuck with more ambitions than time. I haven't kept up with progress since the late '90s, I guess I'm jumping in the deep end here.

Now I'm off to rtfm...

And big thanks to @fohdeesha for this most excellent thread/post/whatever... I'm swedish, no habla ingles...

 

[Zervun]

I'm not sure what I'm getting myself into but this thread was just so great I've ordered three icx6610 from two different sellers... all the way from the states to the north ass-end of Sweden. I guess I should start reading the "fine print" on page one before I get burnt to a crisp for stupid questions. From what I understand the last two I ordered should have atleast one 4-port license upgrade for the sfp ports.

I sort of got into to this because I got tired of my synology-devices being crap. I bought two Dell r720xd, followed by some reading just to realize I can't keep doing 1gbe anymore because... upgrades! So now I'm stuck with more ambitions than time. I haven't kept up with progress since the late '90s, I guess I'm jumping in the deep end here.

Now I'm off to rtfm...

And big thanks to @fohdeesha for this most excellent thread/post/whatever... I'm swedish, no habla ingles...

Page one around post 6 has info for the licenses. You shouldn't need licenses when ordering them. Need to email fodesha

 

[tjk]

ICX6650​

• $300 (or even less occasionally)
• 56x 10gbE (48 via front SFP+, 8 via 2x rear QSFP breakout)
• 4x 40gbE via rear QSFP
• 115w power draw
• a little louder than the icx6610
• all the advanced software features of the icx6610 (vrf's, bgp, tunnels, etc)
• takes same PSU and fan tray models as the ICX6610 (same rack ears too)
• does NOT support stacking
• DOES support MC-LAG (LACP across switches) (Brocades term is MCT)

a little much for most home users, but a good deal if you have a lot of 10gb to aggregate

this seller accepted a BO of $250. It was listed and shown as no rack ears, but came with some. Can't promise his remaining stock will as well - Brocade ICX6650 56-Port 10/40 Gigabit Ethernet Top-of-Rack Switch 42111395699 | eBay


soul extraction:

My secret is out!

Did you get the license "stuff" working with these yet?

 

[fohdeesha]

My secret is out!

Did you get the license "stuff" working with these yet?

yep, hence the post. Wanted to include them in the original post nearly a year ago but they're nearly useless without licenses :/

 

[Emdee]

I'm looking into actually hooking up my Dell R720 to my 6610 and after searching through the many pages in this thread I came to the following conclusion: Everything is a lot more expensive in Europe.

Would buying these two be a smart choice?
Mellanox ConnectX-3 CX354A PCIe x8 40 GBe QSFP+ Dual Port Server HP 649281-B21 | eBay
Brocade 40G-QSFP-4SFP-C-0301 40G QSFP+ Breakout DAC Cable

I can't really find a Mellanox ConnectX-3 cheaper in the Netherlands / EU, same for the cable so I figured I'd get a new one.

Would this be a good combo, or is there a better way to do this?

 

[fohdeesha]

the card looks good, but that cable is an SFP+ breakout cable, when it sounds like you want to do 40gb direct to that mellanox card in your dell, in which case you need qsfp to qsfp like this NEW Brocade 58-0000033-01 4x10GE QSFP Passive Cable 1M | eBay

 

[arglebargle]

are they getting sent to the physical serial port or the telnet/ssh session? Either way probably no way to silence them, on purpose, I can't think why you would ever want to not want to know about a critical piece of the switch failing (other than well, because you replaced the fans and it's wrong )

you could just cut the RPM sense wire, that way it'll think it has failed right off the bat on boot and never report again. Or get a faster fan

They're going to the physical serial port, I have a pi plugged into the same switch that the 6450 is uplinked to so I can faf about with the thing and not lock myself out.

Here's some fan data for anyone shopping for quiet 6450 replacements

https://media.digikey.com/pdf/Data Sheets/Sunon PDFs/MagLev_Motor_Fan.pdf

The KDE1204PKV3 (0.4W) generates intermittent "Fan Up, Fan Down" logs but works fine otherwise.
The MB40201VX (KDE1204PKVX redesign) (1.4W) works fine.

It's tricky sourcing the V1 without paying the full Digikey price but I've got a couple of V2 (0.8W) fans on the way to test too.

 

[i386]

I'm looking into actually hooking up my Dell R720 to my 6610 and after searching through the many pages in this thread I came to the following conclusion: Everything is a lot more expensive in Europe.

Would buying these two be a smart choice?
Mellanox ConnectX-3 CX354A PCIe x8 40 GBe QSFP+ Dual Port Server HP 649281-B21 | eBay
Brocade 40G-QSFP-4SFP-C-0301 40G QSFP+ Breakout DAC Cable

I can't really find a Mellanox ConnectX-3 cheaper in the Netherlands / EU, same for the cable so I figured I'd get a new one.

Would this be a good combo, or is there a better way to do this?

Are you looking for connect-x3 cards just in the netherlands?
If you don't mind to wait you can get them cheaper from the states: HP 544QSFP Infiniband 10/40GB FDR/EN PCIe Dual Port HCA 656089-001 649281-B21 | eBay

 

[arglebargle]

I watched a pair of Sun branded C-X3's go for around $30 this morning as well. Let me find a link to their OEM part number.

This is ended but it has the part numbers you're looking for:
LOT 2 Sun Oracle 7046442 40GB+10Gb Dual QSFP Infiniband Connect PCI-Express Card | eBay

 

[fohdeesha]

I recently purchased a used ICX6610-48P-PE with the PREM and 10G licences. If I update the firmware to latest, will the licences still be valid/unaffected? Do I need to back up the licences prior to updating the firmware?

firmware updates do not affect licenses. and if in some freak accident you lose them, pm me and I'll send more

 

[fohdeesha]

I'm not a networking guy. Two months ago (on a whim) I picked up a MikroTik CRS326 and ConnectX-2 cards, and the SFP+ just works. Shocked, I immediately started looking for more and affordable SFP+ ports. Hence looked at LB6M though bought Brocade icx6450-12p (hoping for lower power consumption).

I go to "Fohdeesha Docs" -- which is mind-bogglingly, take-my-money-please amazing -- and one of the first things is "Connect to the switches serial/console port..."

Umm, ok.

This may be a (very) dumb question, but which is the serial port? I've looked through the 6450 docs and ... there appears to be no labeling, just icons. I'm guessing the top (left-most) port is the serial, and the one just below it is the mgt. port.

The next question then is, since my computer(s) has no serial port, how do I connect?

There's this
USB to RS232 Serial to RJ45 CAT5 Console Adapter Cable for Cisco Routers FTDI | eBay
But is that actually converting anything? [I literally have no COM ports in my (w10) Device Mgr.] So would I instead need a combo RJ45-to-DB9 cable and then a DB9-to-USB cable? Like this

TRENDnet USB to Serial Converter, USB 1.1 to RS-232 Male (9-pin) DB9 Serial | eBay

9-Pin DB9 Serial RS232 Port to RJ45 Cat5 Ethernet LAN Rollover Console 797734232417 | eBay

Thanks in advance. [Already installed PuTTy and Tftp via Chocolatey]

if you bought the ICX 6450-C12, the smaller white thing, be aware it has no 10gb ports - it's not listed in the original post for a reason. The first cable you linked will indeed work, it's a usb to serial converter with the proper rj45 end that our switches have. The serial port is the one labeled with ones and zeroes like "IOIO", the ethernet management port is the one with the network/computer icon

 

[ekv]

if you bought the ICX 6450-C12

D'oh! brain fade, sorry. ICX 6450-24p.

The first cable you linked will indeed work.... The serial port is the one labeled with ones and zeroes like "IOIO", the ethernet management port is the one with the network/computer icon

Bought 'em all yesterday. Figured something would eventually stick to the wall (as it were 8)

serial on top, mgt right below. Got it. Many thanks.

 

[fohdeesha]

added ICX6650 config guide: ICX6650 - Fohdeesha Docs

pm me for lics

 

[fohdeesha]

anymore requests or can I disappear for like 9 years now

 

[svtkobra7]

anymore requests or can I disappear for like 9 years now

Wait who are you again?

(and pls don't => need mo ports already (I bought "the lil' guy") and will need you to answer my rudimentary questions)