Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[EngineerNate]

Hey Guys!

First post here but have been lurking on this thread for a bit now.

I've seen some mention of the 7150s but most say don't go after them because of price. I'm able to grab what's listed as a " ICX7150-24-4X1G" for $76.

Just looking for a basic switch to play with and throw in my rack as my Unifi POE switch is getting pretty full. The lower the power draw and noise the better. I imagine this is similar or better than the 7250 in these categories?

This seems worth it to me. I'm a bit confused on the 4X1G piece. Does this mean that it's not 10G or do I just need a license to unlock them?

Any input would be greatly appreciated.

Thanks!

Buy it. We can probably figure out the license for 10g. That's a great deal.

 

[EngineerNate]

Hey we're at 201...

Dan, sorry for your loss. One of my other hobbies is restoring old woodworking equipment, and the forum for that has a rule:

"In the time between finding a machine and asking others if you should buy it someone else will come to the proper conclusion."

 

[dragonian]

Made it through! 201 pages over 2 days.
Thanks for the informative thread. Thanks @fohdeesha for the commitment over the past 2 years to this little community.

I ordered a ICX6450-24 to consolidate some Mikrotik gear (CRS125 + CRS305) The -P would have been nice, but I couldn't find one with a reasonable price. It's due to arrive on Thursday.

I'm excited to dive into the L3 routing & ACLs and alleviate some of the traffic going up to my OPNsense router.

I think the 6450 is the right size for this house, but as I'm working on planning for a new house, I'm going to have an eye out for future hardware.
I'm thinking that the 7250 will be nice for the more 10gbe SPF+. In total, I'm thinking that I will want up to 48x1Gbe ports, ~10 of those would be PoE, and 5 -7x 10GbE.

Question: Is it better to get 2x 7250-24 and stack them (1 Poe, 1 not).. or 1x 7250-48P. Mostly concerned with power. I don't think i really need HA.
Or would the combo of 2x 6450's be enough?

 

[csementuh]

Anyone know how the power draw is on the ICX6430 models? 24P model perhaps.

I have a 6450-48P with a blown POE board so I did an experiment. Unhooked the POE board and fans and the switch still draws 45-55 watts doing nothing. So not really wanting to leave that on 24/7 as a secondary switch. Will sell for a good price if anyone can use it!

I may have to just break down and buy a 6450-24P or 7250-24P to use for now, then move it to my detached garage next year with a fiber link when done. The 6450-24P says it uses around 25 watts of power which is nice but does the 7250-24P still use 50 watts like my bigger 48 port model does? If so I'll gladly give up the extra SFP+ ports.

 

[StarsAndBars]

Hey everyone!

Just got my 6610 into my rack and have things mostly reconnected. I followed the update guide and things went well. I have a few newbish questions:

1. How do I find out what license(s) this device has, and how do I upgrade if the front ports are not enabled?

2. On the front SFP ports (X1-X8) do I need to issue terminal commands to activate them? If I am using X1 does that mean that Ethernet 1 is now disabled and cannot be used? I have a device I want to connect using a 10GbE copper transceiver that I want to put into X1 instead of using copper ethernet port 1.

3. Regarding the rear connectors, I purchased a DAC breakout cable to connect to other devices. Any commands to enable them? Do I need to manually assign an IP address to them that matches the static IP address assigned to the devices they will connect to? If I use them, does it disable any of the front copper ports by doing so?

Thanks, please advise!

-Rob

 

[LodeRunner]

Show license: sh lic
Show interface information: sh int br
Configuring:
conf t
int eth <number from sh int br, eg. 1/2/1>

For further info see Fohdeesha's guides linked on page 1

 

[bashNinja]

I am not responsible if you fry anything

I made my own serial cable for the ICX 7250 and thought some might appreciate a short guide on how to make them without cutting open cables or soldering. It's silly to spend $50 for an official cable.

Using the pinout, you can easily build your own cable. Use any wire you can find. Ethernet wire works great. You can find cheaper parts elsewhere, but most people want their cable fast, so prime shipping is nice.

Parts, Total: $14.98
$6.99 - Amazon.com: Twinkle Bay DB9 Breakout Connector RS232 Serial 9 Pin Connector Db9 Terminal Female with Screw: Home Audio & Theater
$7.99 - Amazon.com: Kework 11.8 inch USB 2.0 Female to 5 Pin Screw Terminal Female Charging and Data Transfer Adapter Connector Converter Extension Shield Cable Cord (USB Female): Computers & Accessories

Wiring Guide:
D+ to RXD (2)
D- to TXD (3)
Gnd (-) to Gnd (5)
Do not connect anything to the VCC, PWR, +, or VBUS lines

Here's what mine looks like. I used a different USB connector, but it's the same thing.


Then you just connect the DB9 to some serial reader then grab a normal mini usb cable and plug it into the USB breakout and your switch.

Hopefully someone finds that helpful.

 

[StarsAndBars]

Show license: sh lic
Show interface information: sh int br
Configuring:
conf t
int eth <number from sh int br, eg. 1/2/1>

For further info see Fohdeesha's guides linked on page 1

Thanks, but that really didn't answer all of my questions...

What are the various licenses and how do I differentiate between them?

Do the SFP ports on the front and rear conflict with one another? i.e. does copper port 1 become inactive if X1 is used?

Do the front and rear interfaces come up automatically if something is connected or do they have to be manually configured?

 

[Jason Antes]

Thanks, but that really didn't answer all of my questions...

What are the various licenses and how do I differentiate between them?

Do the SFP ports on the front and rear conflict with one another? i.e. does copper port 1 become inactive if X1 is used?

Do the front and rear interfaces come up automatically if something is connected or do they have to be manually configured?

The front and rear ports do not conflict with each other. The 6610 is a beast of a switch and will give you full loving on every port it has.

 

[fohdeesha]

I can't count

 

[Spearfoot]

Hey everyone!

Just got my 6610 into my rack and have things mostly reconnected. I followed the update guide and things went well. I have a few newbish questions:

1. How do I find out what license(s) this device has, and how do I upgrade if the front ports are not enabled?

2. On the front SFP ports (X1-X8) do I need to issue terminal commands to activate them? If I am using X1 does that mean that Ethernet 1 is now disabled and cannot be used? I have a device I want to connect using a 10GbE copper transceiver that I want to put into X1 instead of using copper ethernet port 1.

3. Regarding the rear connectors, I purchased a DAC breakout cable to connect to other devices. Any commands to enable them? Do I need to manually assign an IP address to them that matches the static IP address assigned to the devices they will connect to? If I use them, does it disable any of the front copper ports by doing so?

Thanks, please advise!

-Rob

Most of your questions are answered at @fohdeesha's guide, which gives you detailed instructions for flashing the boot and firmware images on the switch and setting up a basic configuration, among many other things. He also provides a zip file you can download; this contains the boot/firmware images and documentation.

FCX / ICX6610 - Fohdeesha Docs

fohdeesha.com

Think of the 6610 as having three banks of ports: one bank is the 48 x 1G ports on the front panel, another bank is the 8 x SFP/SFP+ ports on the front panel, the last bank is the 4 x 40G ports on the back of the switch (two of which require breakout cables to give you 8 x SFP/SFP+ ports). All three banks are independent of each other and you can use all of the ports in each bank simultaneously, i.e., using a particular port does not disable any other ports.

You can certainly use a 10GBASE-T transceiver in the front SFP/SFP+ ports to attach a copper 10G connection. FWIW, my understanding is that these run hotter than fiber or passive transceivers.

You don't have to configure IP addresses on the ports, though you can configure the switch to provide IP addresses via DHCP, if desired. I don't use the switch for DHCP; I use pfSense. I assign static IP addresses to nearly all of my machines anyway. Basically, you just plug your devices in.

You'll need to connect to the switch either via the serial console or an SSH session (both are covered in @fohdeesha's guide above) to set up the switch, including licensing.

This is a massive thread and there's an immense amount of good information here posted by some smart folks who know what they're doing. I've learned a lot by poring over it, which I've done more than once.

 

[Spearfoot]

I can't count

...the number of times you've had to repeat yourself?

 

[RedX1]

Hi



I am somewhat unfamiliar with the Brocade ICX switches and recently obtained an FCX-648E to explore and learn.

I have upgraded it to the latest firmware and applied the relevant Advantage licences. Now I have a fully operational switch with the 4x10G ports all active.



I am struggling to find a way to gracefully shut this switch down!



I have explored a lot of the documentation and the command menus but have not found any relevant command.

I see from the Brocade documentation for other switches that SysShutdown is used, but that does not work on this switch



Is there any command like the Mellanox - Reload Halt?



I really don’t like just switching off the power at the socket.



Any guidance would be appreciated.





Thanks





RedX1

 

[EngChiSTH]

Made it through! 201 pages over 2 days.
Thanks for the informative thread. Thanks @fohdeesha for the commitment over the past 2 years to this little community.

I ordered a ICX6450-24 to consolidate some Mikrotik gear (CRS125 + CRS305) The -P would have been nice, but I couldn't find one with a reasonable price. It's due to arrive on Thursday.

I'm excited to dive into the L3 routing & ACLs and alleviate some of the traffic going up to my OPNsense router.

I think the 6450 is the right size for this house, but as I'm working on planning for a new house, I'm going to have an eye out for future hardware.
I'm thinking that the 7250 will be nice for the more 10gbe SPF+. In total, I'm thinking that I will want up to 48x1Gbe ports, ~10 of those would be PoE, and 5 -7x 10GbE.

Question: Is it better to get 2x 7250-24 and stack them (1 Poe, 1 not).. or 1x 7250-48P. Mostly concerned with power. I don't think i really need HA.
Or would the combo of 2x 6450's be enough?

Two is more than one (or as the saying goes 'one (of something) is none, two is one, three is right for me').

if you have a single device =>you have a single point of failure. how important is it for you not to have downtime? only you know.

for me, with kids wanting their youtube/Netflix, I rather not hear from about it from them ..

 

[StarsAndBars]

Most of your questions are answered at @fohdeesha's guide, which gives you detailed instructions for flashing the boot and firmware images on the switch and setting up a basic configuration, among many other things. He also provides a zip file you can download; this contains the boot/firmware images and documentation.

FCX / ICX6610 - Fohdeesha Docs

fohdeesha.com

Think of the 6610 as having three banks of ports: one bank is the 48 x 1G ports on the front panel, another bank is the 8 x SFP/SFP+ ports on the front panel, the last bank is the 4 x 40G ports on the back of the switch (two of which require breakout cables to give you 8 x SFP/SFP+ ports). All three banks are independent of each other and you can use all of the ports in each bank simultaneously, i.e., using a particular port does not disable any other ports.

You can certainly use a 10GBASE-T transceiver in the front SFP/SFP+ ports to attach a copper 10G connection. FWIW, my understanding is that these run hotter than fiber or passive transceivers.

You don't have to configure IP addresses on the ports, though you can configure the switch to provide IP addresses via DHCP, if desired. I don't use the switch for DHCP; I use pfSense. I assign static IP addresses to nearly all of my machines anyway. Basically, you just plug your devices in.

You'll need to connect to the switch either via the serial console or an SSH session (both are covered in @fohdeesha's guide above) to set up the switch, including licensing.

This is a massive thread and there's an immense amount of good information here posted by some smart folks who know what they're doing. I've learned a lot by poring over it, which I've done more than once.

I actually have read his instructions and followed them to update my 6610, so I am definitely thankful for that. I did establish a serial connection to configure it. For example, I was having a PoE issue where a device was not being powered. I didn't realize that apparently after the firmware upgrades in his guide, all of the PoE is disabled? I had to find out how to manually enable inline power on the switchports I needed PoE on. What the guide did not cover (and if it did, I missed it) is:

1. Confirmation anywhere that the three banks of ports are indeed independent of one another, so thanks for clearing that up.

2. I put a 10GbE transceiver that I know is good and works elsewhere into one of the front SFP ports and then I connected a 10GbE device to it. It does not light up and it does not link on the device. What do I need to do to resolve that?

3. I bought a DAC breakout cable. I tried connecting some SFP-capable devices to it, but again, no link/connection. Where in the documentation does it cover the commands necessary (if any) to activate those ports? I saw the blurb about activating the trunk ports, but I don't yet have a second 6610 to link so are there other commands I need to use in the interim?

4. Licensing. I want to get a second 6610, so how do I query the one I have and a potential seller of a second to make sure that everything is what it needs to be to enable the full capabilities of the switch? A primer on how to interpret what the switch reports back when queried would be particularly helpful. Or, does it not matter? Are all 6610s that are in working order good to go and will not require further intervention.

Again, I don't mind reading, and I have read through a large number of the posts here. I just don't think it is realistic to expect someone to pour through 200+ pages of posts to pluck out what they need. I'd be happy to contribute by writing a FAQ if someone would be willing to help me pull together the topics.

Thanks again!

 

[Spearfoot]

2. I put a 10GbE transceiver that I know is good and works elsewhere into one of the front SFP ports and then I connected a 10GbE device to it. It does not light up and it does not link on the device. What do I need to do to resolve that?

What brand of transceiver are you using? These switches seem to work with just about anything you throw at them, but perhaps not with your transceiver. You can use the 'show media' command to see how the switch 'sees' your transceiver. The front panel SFP/SFP+ ports are 1/3/1-8; here's the output from my port 1/3/2, where I've installed a FiniSar transceiver:

SSH@brocade-6610-u1.spearfoot.net#show media ethernet 1/3/2
Port   1/3/2: Type  : 10GE SR 300m (SFP +)
             Vendor: FINISAR CORP.      Version: B
             Part# : FTLX8571D3BNL-E5   Serial#: AMQ0TC3

To connect at 10G, you will need to insure the port is configured correctly in your startup configuration. Here's the section of mine for port 1/3/2:

!
interface ethernet 1/3/2
port-name Trunk10G 2
speed-duplex 10G-full
!

And you can get very detailed information about the port state with the 'show interface' command:

SSH@brocade-6610-u1.spearfoot.net#show interface ethernet 1/3/2
10GigabitEthernet1/3/2 is up, line protocol is up
  Port up for 7 day(s) 9 hour(s) 51 minute(s) 56 second(s)
  Hardware is 10GigabitEthernet, address is 748e.f8f6.f700 (bia 748e.f8f6.f73c)
  Interface type is 10Gig SFP+
  Configured speed 10Gbit, actual 10Gbit, configured duplex fdx, actual fdx
  Member of 4 L2 VLANs, port is tagged, port state is FORWARDING
  BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
  Link Error Dampening is Disabled
  STP configured to ON, priority is level0, mac-learning is enabled
  Openflow is Disabled, Openflow Hybrid mode is Disabled,  Flow Control is enabled
  Mirror disabled, Monitor disabled
  Mac-notification is disabled
  Not member of any active trunks
  Not member of any configured trunks
  Port name is Trunk10G 2
  MTU 10200 bytes, encapsulation ethernet
  300 second input rate: 15944 bits/sec, 9 packets/sec, 0.00% utilization
  300 second output rate: 38272 bits/sec, 15 packets/sec, 0.00% utilization
  194948462 packets input, 1416794689861 bytes, 0 no buffer
  Received 234556 broadcasts, 124805 multicasts, 194589101 unicasts
  0 input errors, 0 CRC, 0 frame, 0 ignored
  0 runts, 0 giants
  306093793 packets output, 810296349818 bytes, 0 underruns
  Transmitted 588139 broadcasts, 57908 multicasts, 305447746 unicasts
  0 output errors, 0 collisions
  Relay Agent Information option: Disabled

Egress queues:
Queue counters    Queued packets    Dropped Packets
    0           306066113                   0
    1                   0                   0
    2                   0                   0
    3                   0                   0
    4                   0                   0
    5               14140                   0
    6                   0                   0
    7               13540                   0

3. I bought a DAC breakout cable. I tried connecting some SFP-capable devices to it, but again, no link/connection. Where in the documentation does it cover the commands necessary (if any) to activate those ports? I saw the blurb about activating the trunk ports, but I don't yet have a second 6610 to link so are there other commands I need to use in the interim?

The 10G breakout ports are 1/2/2-5 and 1/2/7-10, ports 1/2/1 and 1/2/6 are the 40G ports. You can check the media state for these with the 'show media' command at the console. Here's output from my switch, showing that I have an Arista breakout cable attached:

SSH@brocade-6610-u1.spearfoot.net#show media ether 1/2/2
Port   1/2/2:Type  : 40GBASE-Passive Copper
Vendor Name: Arista Networks  Serial Num: XHC1314LG06P    Revision: 01
SSH@brocade-6610-u1.spearfoot.net#show media ether 1/2/3
Port   1/2/3:Type  : 40GBASE-Passive Copper
Vendor Name: Arista Networks  Serial Num: XHC1314LG06P    Revision: 01
SSH@brocade-6610-u1.spearfoot.net#show media ether 1/2/4
Port   1/2/4:Type  : 40GBASE-Passive Copper
Vendor Name: Arista Networks  Serial Num: XHC1314LG06P    Revision: 01
SSH@brocade-6610-u1.spearfoot.net#show media ether 1/2/5
Port   1/2/5:Type  : 40GBASE-Passive Copper
Vendor Name: Arista Networks  Serial Num: XHC1314LG06P    Revision: 01
SSH@brocade-6610-u1.spearfoot.net#

4. Licensing. I want to get a second 6610, so how do I query the one I have and a potential seller of a second to make sure that everything is what it needs to be to enable the full capabilities of the switch? A primer on how to interpret what the switch reports back when queried would be particularly helpful. Or, does it not matter? Are all 6610s that are in working order good to go and will not require further intervention.

Licensing is a little complicated on these devices, but typically the 'show license' command will return something like this:

SSH@brocade-6610-u1.spearfoot.net#show license
Index    Lic Mode        Lic Name               Lid/Serial No  Lic Type    Status     Lic Period    Lic Capacity
Stack unit 1:
1        Node Lock       ICX6610-PREM-LIC-SW    dzmHKHMlFmK    Normal      Active      Unlimited         65535
2        Node Lock       ICX6610-10G-LIC-POD    dzmHKHMlFmK    Normal      Active      Unlimited         4

In this case, the switch has a premium software license and the first 4 front-panel SFP ports are licensed for 10G, so the last 4 will only run at 1G. Full details about model numbers and licensing are available in the datasheet: https://fohdeesha.com/data/other/brocade/ICX6610 Datasheet.pdf

In a nutshell, the model numbers have a schema like this: ICX 6610-[ports][poe]-[premium_software][airflow] where:

• ports: 24 or 48 (will include 'F' suffix if these ports are SFP ports)
• poe: 'P' if the unit provides PoE, blank otherwise
• premium_software: 'P' if present, blank otherwise
• airflow: 'E' = power-supply-side exhaust, 'I' = power-supply-side intake

Both of my switches are model ICX 6610-48P-PE -- they have 48 ports providing PoE; have the premium software; have a license for 4 x 10G SFP+ front-panel ports; and airflow is from front-to-back.

Hopefully this arms you with enough information to nail down the type of switch you want to purchase.

This post has gotten incredibly long and complicated, so I'm going to sign off with a final note that, while pretty clunky, the 6610 web GUI is useful for both investigating and configuring these switches.

 

[StarsAndBars]

EXACTLY what I was looking for. Thank you! To confirm, if the listing mentions a PREMIUM license, that means that X1-4 are 10GbE enabled?

 

[Spearfoot]

EXACTLY what I was looking for. Thank you! To confirm, if the listing mentions a PREMIUM license, that means that X1-4 are 10GbE enabled?

Yes, sir

 

[fohdeesha]

As the OP of this thread says, pm me for licenses to unlock your 10gb ports

 

[fohdeesha]

Yes, sir

Premium is the name of one of the advanced routing licenses, doesn't have anything to do with the 10gb ports. The 10gb ports use a PoD license