Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[ManoftheSea]

@Gerhen
First of all, in neither config does it look like you've assigned VEs to your VLANs. Within your VLAN configurations, you need to issue "router-interface ve N" statements.

I'm confused by your statement "router-on-a-stick" but also "switch handling inter-vlan". I understand inter-VLAN to mean doing the routing between VLANs, where "intra-VLAN" would be L2 switching.

Where are other devices plugged in? The router on a stick is in 1/2/1?

 

[Gerhen]

@Gerhen
1. First of all, in neither config does it look like you've assigned VEs to your VLANs. Within your VLAN configurations, you need to issue "router-interface ve N" statements.

2. ...I'm confused by your statement "router-on-a-stick" but also "switch handling inter-vlan"....

3. Where are other devices plugged in? The router on a stick is in 1/2/1?

Itemizing my reply to help with readability:

​

1. I’m on the new 09.0.10 version of fastiron and they've removed the "router-interface ve N" statements. In this version, when you create VLAN XX, a corresponding "interface ve XX" is created where XX matches between VLAN and Interface.​

(with the 08x firmware, the matching between VLAN number to the Interface number was not a hardset rule, so they had "router-interface" there to point to the correct interface for that vlan)​

​

2. Regarding "router-on-a-stick", you're right! I misspoke, I've edited my prior post for clarity. I will have a "one-arm" trunk port between the firewall and the L3 router but the L3 router will handle all the internal routing (there's no L2 switch in the topology which would be the case otherwise).​

​

3. At this time only using two connections: 1/2/1 which connects to the firewall and 1/1/1 which is connected to one PC so that I can test that DHCP is working. I'm carefully provisioning the Brocade unit alongside my existing network in the hope that I can swap the new hardware in with minimal downtime.​

 

[ManoftheSea]

1. Ah, I see. My fault, I didn't know the newer firmware did so. It's a sensible change, I'll keep it in mind for future discussion.
2. I've got a setup where my Internet Gateway Device (OpenWRT on an EspressoBIN) has a point to point connection. I don't send all the VLANs to it. If I did, I would need to tell it to expect the tagged traffic on sub-interfaces of the ethernet port. So, for your situation where only TAGGED 255 traffic works, I would want to look at the pfSense and see if it's expecting the traffic to be tagged.
3. Based on the connections, you said DHCP works when you tag VLAN 225? I see you have the helper-address set for ve/vlan 20 and that's untagged to the PC, which is all as I would expect. No DHCP snooping, so no DHCP trust assignment either.

It's not relevant to your current problem, but what's the reason behind sending all the VLANs to this device?

 

[TeleFragger]

Brocade ICX6610 - ~$200 on ebay

​

• the BEEF KING
• 24/48 1gbE copper (PoE available)
• 16x 10gbE (8x SFP+ in the front, 8x via 2 QSFP+ breakout ports on the rear)
• 2x 40gbE (separate from the previously mentioned breakout ports)
• Supports OpenFlow in hardware for SDN, including hybrid port mode
• SupportS MACSEC on the SFP+ ports for 80gbps of real time L2 AES-128 encryption
• Same OS features as ICX6450 but adds advanced protocols like BGP, VRFs, tunnels, everything
• 80w power draw for the 24 port models with or without PoE
• 110w power draw for the 48 port models with or without PoE
• audible - about the same as an R710, little quieter than LB6M
• 2x redundant hot-swap PSU's
• Fans cannot be modified
• Aggregate capacity: 528gbps / 396Mpps (wirespeed regardless of features enabled)
• 
  
• Datasheet
• FAQ
• Architecture Brief
• (note: when the above PDF's say the QSFP ports can only be used for stacking, they're lying)

I am looking at one of these on ebay and I am just asking a few questions here to see if this gets me what I need.
I currently have a Brocade FastIron FCX648s and it works fine. I have an adapter in the back for 10gb (old cx4 connector they call it)

This thing is catching my eye for a few things...
1 - 10gbE - convert my machines from the old connectx-1 cx4 thick cable to sfp+
I have Reolink POE cameras and I see it says PoE Available.

So My questions:

1. What is needed to do POE on these? so I can remove my 8 port POE switch and use this switch?
2. do all brocade adapters from other switches work on each other model? so I can move my CX4 10gb adapter over?
3. Found one on ebay (he has 8 left) and says SWITCH HAS BEEN RESET, NO SOFTWARE OR LICENSING WILL BE INCLUDED!
   1. My question is what the licenses is used for as i am just doing this as a home user that has a home virtual lab, etc. and not using any in switch features.

I think that is good to start with. thanks for any help.

 

[bwahaha]

So My questions:

1. What is needed to do POE on these? so I can remove my 8 port POE switch and use this switch?
2. do all brocade adapters from other switches work on each other model? so I can move my CX4 10gb adapter over?
3. Found one on ebay (he has 8 left) and says SWITCH HAS BEEN RESET, NO SOFTWARE OR LICENSING WILL BE INCLUDED!
   1. My question is what the licenses is used for as i am just doing this as a home user that has a home virtual lab, etc. and not using any in switch features.

1) make sure it's a poe model. There are some without.
2) sfp module? mine doesn't care about brand, so yes.
3) Follow the guides for the 6610, all licensing will be applied. You want the licensing for 10gbe.

 

[TeleFragger]

1) make sure it's a poe model. There are some without.
2) sfp module? mine doesn't care about brand, so yes.
3) Follow the guides for the 6610, all licensing will be applied. You want the licensing for 10gbe.

so i want the licensing... how much does that cost?
here is what i am looking at..
Brocade ICX6610-48P-E 48-port PoE+ Gigabit Ethernet Switch 8x 10GbE 1xPSU/Fan | eBay


Edited - think i found it.. they are not 10gb.. but 1gb sfp+ ports?
side note, i just noticed that my current switch has 4 sfp+ ports??? i just found a random cable from my hoarding (hah) and it fits in there..
so if i get an sfp+ card, I can just run with that or as you are saying licensing?
I know nothing about these switches, but that someone here recommended the one I have as I found it years ago with the needed port on the back to connect to my hp procurve 6400cl!

 

[bwahaha]

so i want the licensing... how much does that cost?
here is what i am looking at..
Brocade ICX6610-48P-E 48-port PoE+ Gigabit Ethernet Switch 8x 10GbE 1xPSU/Fan | eBay

side note, i just noticed that my current switch has 4 sfp+ ports??? i just found a random cable from my hoarding (hah) and it fits in there..
so if i get an sfp+ card, I can just run with that or as you are saying licensing?
I know nothing about these switches, but that someone here recommended the one I have as I found it years ago with the needed port on the back to connect to my hp procurve 6400cl!

Follow the guide, licensing is applied for "free"; you pay nothing. Good seller, imo. I got mine from them.

I won't comment on licensing for the fcx, since I don't know.

https://forums.servethehome.com/index.php?threads/brocade-icx-series-cheap-powerful-10gbe-40gbe-switching.21107/page-143#post-248899

Bit more info in the link to the 648s.

First there's the "stacking" FCX models: These are the FCX624S and the FCX648S (the models with "S" at the end). These are by far the most popular and easiest to find for pennies. These have two special CX4 16gbps ports on the rear to stack with each other. These "stacking" models only take the 2-port 10gbE XFP card. Search ebay for "ES4625M" to find these cards around $40. This is the easiest option if you want a good lab switch with 2x 10gbE ports for around $70 total.

 

[TeleFragger]

Follow the guide, licensing is applied for "free"; you pay nothing. Good seller, imo. I got mine from them.

I won't comment on licensing for the fcx, since I don't know.

https://forums.servethehome.com/index.php?threads/brocade-icx-series-cheap-powerful-10gbe-40gbe-switching.21107/page-143#post-248899

Bit more info in the link to the 648s.

My ultimate goal is to "cleanup" my setup as I now have:
brocade 48 port rack mount
hp procurve 6 port rack mount
8 port poe for cameras

Goals are:
15 x1gb ports
6 or more POE
6-8 10 gig ports.. all close to each other except 1 about 15ft away (gaming rig)

do you recommend a different switch or could I do all of this with that switch?

Googled and found specs here Brocade ICX 6610-48P Switch | DataSwitchWorks.com
you mention license is free for 10gb.. i see this..
Dual-mode 1/10 GbE SFP/SFP+ ports
(10 GbE SFP+ optional upgrade license)

 

[els]

I've been reading this thread a bit and looked at different ICX models. I seem to have narrowed to 7150 ZP. Currently I have Juniper EX2200 which has served well for the last 10 years but has some limitations so I am looking to replace it with something better.

Requirements:
1 / 2.5GbE ports (I have Ruckus R650)
POE capability (to power IP cameras, Ruckus AP, as well as 5- or 8-port switches in various rooms)
At least 6 GbE SFP+ (I have 5 10GbE copper which means I would need adapters these are connected directly to servers / my desktop so would uplink ports allow for VLAN, intervlan routing rules, etc?)
L2 / L3 functionality
Furthermore, I read a bit about ICX it sounds like REST API is supported?

Would 7150 ZP fit the bill or should I look for something else? How does 7150 compare to Juniper EX2200 in terms of functionality and performance?

Thanks.

 

[bacourt]

Hey all,

I have a stack of two 6610's and they work great. I was looking at installing certificates from my root CA as I've been working with certificates and it's good practice.

Anyway, after struggling with it for a bit,

*edit* I needed to use a PKCS#1 certificate instead of PKCS#8, I believe? My CA is a Windows Machine, the root CA was added to the bottom of the certificate in base64 format (.pem was used for the certificate and private key, the private key was decrypted).

I finally got the certificate installed - the http access works as intended, you can log in and the page displays correctly, but when attempting to access via https (via IP or hostname) I'm getting this error:

Via Chrome or any other browser.

This site can’t be reached
The webpage at

https://1.2.3.4/

might be temporarily down or it may have moved permanently to a new web address.
ERR_SSL_BAD_RECORD_MAC_ALERT

Turning on http allows me to access the web page, so I'm wondering what's going on here. Searching this error didn't find me much.

Any advice or information is greatly appreciated!

 

[ManoftheSea]

when attempting to access via https (via IP or hostname) I'm getting this error:

Complete guess? Your certificate uses some algorithm the ICX doesn't. Too big RSA key? SHA256 signature?

 

[bacourt]

Complete guess? Your certificate uses some algorithm the ICX doesn't. Too big RSA key? SHA256 signature?

I made sure the RSA bit count was 2048 - do you know what the limit on the hashing signature is instead of SHA256? Any information would be really appreciated

 

[NablaSquaredG]

yeah, but I'm willing to escalate this, possibly to the uboot authors themselves.

I don't care about their support contract shitshow, GPL is GPL

BTW, this has now been escalated to denx who maintain u-boot

 

[CTurtle]

Hey,

First off, the obligatory thank you to fohdeesha et al, it is amazing the effort applied here to avoid e-waste and help the community out, it is highly appreciated and does make one "do better" overall.

With that off my chest, I bought a couple of 6610PE units from ebay, followed the documentation to upgrade and license said units, it all went smoothly as one could expect. Only one is in operation currently, with a handful of 10GB, 1GB and even one of the 40GB ports in use and, until recently, one of the 1GB ethernet ports was supplying POE power to an Ubiquity AP. This unit sits in a rack powered by a single PSU supplied through an HP UPS, so some protection exists.

The other day there was a storm and some ridiculously close lightning strike happen that killed power to the house. We were left in the dark for about one hour and the UPS did turn off, which it really shouldn't so there might be some issue there, but I haven't been able to address that yet. Anyway, everything seemed to power up fine, there seems to be zero damage which, frankly, is quite astonishing but I do have surge suppressors everywhere and good GFCIs all around.

What I noticed yesterday is that the AP was out, so I though "there, that's one victim to the storm" but testing it using a separate POE injector worked fine, so I thought maybe I had forgotten to "write mem" when I turned on the inline power, but no, it was still set up to be on:

SSH@puft-prince>show inline power 1/1/13

 Port   Admin   Oper    ---Power(mWatts)---  PD Type  PD Class  Pri  Fault/
        State   State   Consumed  Allocated                          Error
--------------------------------------------------------------------------
 1/1/13 On      Off            0          0  n/a      n/a         3  n/a

SSH@puft-prince>show inline power

Power Capacity:         Total is 748000 mWatts. Current Free is 748000 mWatts.

Power Allocations:      Requests Honored 0 times
...

I tried ports in each of the 4 16 port groups, but clearly something broke here. The switch is otherwise working great still, so I wondered if, me being handy with the soldering iron and all, I should even attempt to troubleshoot this, and if there are any pointers you could provide me as to how to proceed.

Or maybe I just downgrade the switch to a non POE version in my head, and use it as such

Thanks in advance.

 

[jode]

Check the status of your POE module.

SSH@ICX6610#show chassis
The stack unit 1 chassis info:

Power supply 1 (AC - PoE) present, status ok
     Model Number:    23-0000141-02
    Serial Number:    61L     
    Firmware Ver:      C
...

and

SSH@ICX6610#show module
       Module                                         Status Ports Starting MAC 
U1:M1  ICX6610-24P POE 24-port Management Module        OK     24   78a6.e144.eede
U1:M2  ICX6610-QSFP 10-port 160G Module                 OK     10   78a6.e144.eede
U1:M3  ICX6610-8-port Dual Mode(SFP/SFP+) Module        OK     8    78a6.e144.eede

If the results say anything other than 'OK' you need to consider downgrading it to a non POE switch, or fix the hw.

 

[CTurtle]

Check the status of your POE module.

Looks fine, right?

SSH@puft-prince>show chassis
The stack unit 1 chassis info:

Power supply 1 (AC - PoE) present, status ok
        Model Number:   23-0000142-02
        Serial Number:  RT8
        Firmware Ver:    A
Power supply 1 Fan Air Flow Direction:  Front to Back
Power supply 2 not present

Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 not present
...

SSH@puft-prince>show module
       Module                                         Status Ports Starting MAC
U1:M1  ICX6610-48P POE 48-port Management Module        OK     48   cc4e.243d.2e4e
U1:M2  ICX6610-QSFP 10-port 160G Module                 OK     10   cc4e.243d.2e4e
U1:M3  ICX6610-8-port Dual Mode(SFP/SFP+) Module        OK     8    cc4e.243d.2e4e

 

[jode]

Looks good to me.

Also, check if the show log command brings up anything related/useful.

Next, I'd try connecting your AP using another cable, to another port, again monitoring activity with show log.

 

[blunden]

First off, the obligatory thank you to fohdeesha et al, it is amazing the effort applied here to avoid e-waste and help the community out, it is highly appreciated and does make one "do better" overall.

Agreed.

What I noticed yesterday is that the AP was out, so I though "there, that's one victim to the storm" but testing it using a separate POE injector worked fine, so I thought maybe I had forgotten to "write mem" when I turned on the inline power, but no, it was still set up to be on [...]

It could also be that the lightning strike fried the PoE negotiation circuitry of the AP. The Unifi PoE injectors was at least in the past passive PoE (i.e. they just inject power, no 802.3af/at/bt negotiation), so they tend to work even when the PoE circuitry fails.

 

[CTurtle]

Looks good to me.

Also, check if the show log command brings up anything related/useful.

Next, I'd try connecting your AP using another cable, to another port, again monitoring activity with show log.

Cables are not the issue, first thing I did was connect the AP to the POE injector using a whole new set of cables, and when that worked out I reuse the same known good cables to try in the switch again, which failed.

Show log show nothing, really. I have both ports 13 and 14 with POE powered devices, and these don't even show up in the log after a cold boot:

SSH@puft-prince#show inline power

Power Capacity:         Total is 748000 mWatts. Current Free is 748000 mWatts.

Power Allocations:      Requests Honored 0 times


 Port   Admin   Oper    ---Power(mWatts)---  PD Type  PD Class  Pri  Fault/
        State   State   Consumed  Allocated                          Error
--------------------------------------------------------------------------
 1/1/13 On      Off            0          0  n/a      n/a         3  n/a
 1/1/14 On      Off            0          0  n/a      n/a         3  n/a

Jan 21 14:02:46:I:System: Interface ethernet 1/1/47, state up
Jan 21 14:02:45:I:System: Interface ethernet 1/1/23, state up
Jan 21 14:02:45:I:System: Stack unit 1 POE  Power supply 1  with 748000 mwatts capacity is up
Jan 21 14:02:45:I:System: Interface ethernet 1/1/24, state up
Jan 21 14:02:45:I:System: Interface ethernet 1/2/2, state up
Jan 21 14:02:45:I:System: Interface ethernet 1/1/15, state up
Jan 21 14:02:43:I:System: Interface ethernet 1/3/3, state up
Jan 21 14:02:43:I:System: Interface ethernet 1/3/1, state up
Jan 21 14:02:43:I:System: Interface ve 1, state up
Jan 21 14:02:43:I:System: Cold start
Jan 21 14:02:17:I:System: Port init success Stack unit 1 Port 1/2/1 Lane 0 T 0 R 0 Type 0:  00x00000x0000 00x00000x0000 00x00000x0000 00x00000x0000 00x00000x0000 00x00000x0000 00x00000x0000 00x00000x0000 00x00000x0000 00x00000x0000 00x00000x0000 00x00000x0000 00x00000x000
Jan 21 14:00:15:I:DHCPC: protocol disabled by user
Jan 21 14:00:15:I:NTP: The system clock is not synchronized to any time source.
Jan 21 14:00:15:I:NTP: The system clock is not synchronized and does not have a reference configured.

It could also be that the lightning strike fried the PoE negotiation circuitry of the AP. The Unifi PoE injectors was at least in the past passive PoE (i.e. they just inject power, no 802.3af/at/bt negotiation), so they tend to work even when the PoE circuitry fails.

Well, I'm embarrassed I didn't think of that immediately, it is a good part of my day job to troubleshoot similar things, but unfortunately that was not the culprit.

I used a Netgear GS108PEv3 switch I had laying around as a POE source, it is supposed to only do 802.3af, so no passive POE in theory, and the AP worked fine connected to it. As a further step I connected another POE powered device (A DLink switch that does POE passthrough) which I have working in my home lan powered by POE, so definitely working correctly, and did a cold boot of the brocade with the switch on port 13 and the AP on port 14... neither got detected by the switch.

At this point the only thing I know for certain is that the issue is within the brocade switch itself.

 

[mp.]

So this might be a dumb question, but does anyone know if the ICX 7150-C08P has 10g sfp+ uplink ports? There is almost no info out there on them because they were discontinued so quickly.