Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[AgentXXL]

will the ICX-6610 function/negotiate to 2.5Gbps speed with a sfp+ RJ45 module?

Only if you can find the specific SFP+ module that's known to work, specifically the Supermicro AOM-AQS-107-B0C2-CX. Be careful - there are many knock-offs that claim to be 100% compatible but all that I've found all use the Marvell chip vs the Supermicro which uses the Aquantia AQS-107. It appears to have a larger buffer which some report works properly with switches like the 6610 that don't specifically allow any SFP+ ports to link at 2.5 or 5Gbps. See this post for more details:

https://forums.servethehome.com/index.php?threads/gbase-t-transceivers-performance-with-nbase-t-marvell-88x3300-v-s-aquantia-aqs-107.30004/

 

[AgentXXL]

I'm having no luck finding the Supermicro AOM-AQS-107-B0C2-CX in stock from a supplier that's based in or will ship to Canada. I'm looking at other options to get the onboard 5Gbps NIC on one of my systems working properly with my 6610. As the 6610 SFP+ ports only link at 1Gbp or 10Gbps, would I be able to use a Mikrotik CRS305 as a rate converter? I.E. attach my 5Gbps via my current Wiitek SFP+ module to one SFP+ port and then use a DAC cable to connect another SFP+ port to one on the 6610? I would be using the Mikrotik with SwitchOS instead of RouterOS.

The Mikrotik will actually work out cheaper than the Supermicro module, but I'm leary about wasting my time ordering one without knowing if it will actually work. Has anyone done this? TIA!

 

[RobstarUSA]

I'm having no luck finding the Supermicro AOM-AQS-107-B0C2-CX in stock from a supplier that's based in or will ship to Canada. I'm looking at other options to get the onboard 5Gbps NIC on one of my systems working properly with my 6610. As the 6610 SFP+ ports only link at 1Gbp or 10Gbps, would I be able to use a Mikrotik CRS305 as a rate converter? I.E. attach my 5Gbps via my current Wiitek SFP+ module to one SFP+ port and then use a DAC cable to connect another SFP+ port to one on the 6610? I would be using the Mikrotik with SwitchOS instead of RouterOS.

The Mikrotik will actually work out cheaper than the Supermicro module, but I'm leary about wasting my time ordering one without knowing if it will actually work. Has anyone done this? TIA!

I'm actually doing this with a CRS317 (but 2.5Gbit/s nics, not 5). There is a LARGE learning curve for Mikrotik. I say this as someone who has 9 years of experience on probably 20-30 Cisco layer3 products -- including configuring & maintaining multicast and 4-5 different routing protocols. I did figue it out but RouterOS was the hardest thing I've had to learn in the networking world.

Also: Be away to ONLY GET THE MIKROTIK MGIG SFPS, or you will run into similar issues you are already having, because 3rd party mgig sfps don't report their speed propertly to the CRS317.

 

[Rain]

As the 6610 SFP+ ports only link at 1Gbp or 10Gbps, would I be able to use a Mikrotik CRS305 as a rate converter? I.E. attach my 5Gbps via my current Wiitek SFP+ module to one SFP+ port and then use a DAC cable to connect another SFP+ port to one on the 6610? I would be using the Mikrotik with SwitchOS instead of RouterOS.

Using another switch in this fashion will definitely work, though you'll want to verify that the SFP+ ports support linking up & functioning at 2.5Gb/5Gb, otherwise you'll be in the same boat. Check the specifications before you buy!

It may be better & more future-proof to just purchase a switch that has 10Gb SFP+ uplinks and native RJ-45 NBASE-T ports if you need NBASE-T functionality and plan on using it for a few years.

 

[AgentXXL]

Also: Be away to ONLY GET THE MIKROTIK MGIG SFPS, or you will run into similar issues you are already having, because 3rd party mgig sfps don't report their speed propertly to the CRS317.

Is that the Mikrotik S+RJ10 module? Unfortunately that bumps the price of a solution back into the same range as one of the Supermicro SFP modules that would work with the 6610 @ 5Gbps. Of course the Mikrotik is a lot easier to find than the Supermicro module so perhaps that's what I'll have to do.

Using another switch in this fashion will definitely work, though you'll want to verify that the SFP+ ports support linking up & functioning at 2.5Gb/5Gb, otherwise you'll be in the same boat. Check the specifications before you buy!

It may be better & more future-proof to just purchase a switch that has 10Gb SFP+ uplinks and native RJ-45 NBASE-T ports if you need NBASE-T functionality and plan on using it for a few years.

Suggestions on a switch that does this without 'breaking the bank'? My understanding is that the CRS305 has 4 SFP+ ports that support 1, 2.5, 5 and 10 Gbps. My current Wiitek SFP module is seen by the 6610 as connecting at 10Gbps, but the system itself reports a 5Gbps link. I suspect I might run into the same issue using it on the CRS305. I purchased the Wiitek as it's affordable, readily available and according the this STH review, it supports 2.5Gbps and 5Gbps NBase-T. It just doesn't have enough buffer to accept the 10Gbps rate that the 6610 thinks it is, even if I rate limit the 6610 SFP port to less than 5Gbps.

https://www.servethehome.com/wiitek-sfp-10g-t-review-10gbase-t-adapter/

I will be using this setup for a few years, but I'm also on a limited budget so I'd like to get the 5Gbps NIC working as affordably as possible. That's why I chose to purchase one of the 6610's, which is working fine for my other 10Gbps and 1Gbps devices.

 

[noduck]

Suggestions on a switch that does this without 'breaking the bank'?

I am using a Netgear MS510TX for this purpose; it works really well for multi-Gig, several RJ45 with various combinations of speeds and one SFP+. It was $210 when I bought it 2 years ago (I cannot find it for sale now). I bought it before I got any of the Brocade switches.

Of course, it does not compare on features with either Mikrotik or Brocade, and only has web UI.

 

[blademan]

Suggestions on a switch that does this without 'breaking the bank'? My understanding is that the CRS305 has 4 SFP+ ports that support 1, 2.5, 5 and 10 Gbps. My current Wiitek SFP module is seen by the 6610 as connecting at 10Gbps, but the system itself reports a 5Gbps link. I suspect I might run into the same issue using it on the CRS305. I purchased the Wiitek as it's affordable, readily available and according the this STH review, it supports 2.5Gbps and 5Gbps NBase-T. It just doesn't have enough buffer to accept the 10Gbps rate that the 6610 thinks it is, even if I rate limit the 6610 SFP port to less than 5Gbps.

https://www.servethehome.com/wiitek-sfp-10g-t-review-10gbase-t-adapter/

I will be using this setup for a few years, but I'm also on a limited budget so I'd like to get the 5Gbps NIC working as affordably as possible. That's why I chose to purchase one of the 6610's, which is working fine for my other 10Gbps and 1Gbps devices.

I’m using a CRS305 and one of these “FLYPROFiber 10GBase-T SFP+ to RJ45 Transceiver, 10Gb SFP+ to RJ45 Ethernet Copper Module for MikroTik S+RJ10, CAT6A/CAT7, 100FT(30M)” based on STH YouTube and review. Negotiates at 2.5Gb, and in Amazon questions section Flypro says it will do NbaseT.

 

[richtj99]

If I want to use the Brocade 7250 for routing (or as a router)? Whats the best way to do that?

Currently I have:
Router - Unifi switch - 1gb fiber to Brocade 7250 10gb fiber to 2nd Brocade 7250 10gb fiber (multiple 10gb devices within).

With multiple vlans in place, the router is slowing all intervlan traffic to 1gb right?

How Do I swap it:

Router - Brocade 7250 10gb fiber to 2nd Brocade 7250 10gb fiber with 1gb fiber connection to Unifi?

Can the Brocade do the routing to keep the 10gb items at 10gb?

 

[richtj99]

As a followup -

does each Brocade switch on each vlan need a dedicated VE number and IP address for internal routing?

Brocade 7250 #1:
Vlan 10
VE 10- 192.168.1.250
Vlan 20
VE 20- 192.168.2.250
Vlan 30
VE 30- 192.168.3.250


Brocade 7250 #2:
Vlan 10
VE 10- 192.168.1.251
Vlan 20
VE 20- 192.168.2.251
Vlan 30
VE 30- 192.168.3.251

Brocade 7250 #3:
Vlan 10
VE 10- 192.168.252
Vlan 20
VE 20- 192.168.2.252
Vlan 30

In the above - would the brocades handle all internal traffic on each Vlan due to the VE interfaces?

Vlan 10 - All 3 switches would route on a layer 3 level?
Vlan 20 - All 3 switches would route on a layer 3 level?
Vlan 30 - Only #1 & #2 would route on a layer 3 level? #3 would not due on Vlan 30 due to no VE 30 interface? #3 would route on layer 3 for Vlan 10 & 20?

Or am i thinking about this in the wrong direction?

 

[AgentXXL]

I’m using a CRS305 and one of these “FLYPROFiber 10GBase-T SFP+ to RJ45 Transceiver, 10Gb SFP+ to RJ45 Ethernet Copper Module for MikroTik S+RJ10, CAT6A/CAT7, 100FT(30M)” based on STH YouTube and review. Negotiates at 2.5Gb, and in Amazon questions section Flypro says it will do NbaseT.

Thanks! I've ordered a CRS305 and one of the Mikrotik S+RJ10 SFP+ modules just in case my Wiitek doesn't work. I'll hopefully have success later this week when they arrive.

In the meantime I've also discovered that the on-board Aquantia 5Gbps NIC (AQC111C) on my Asus Prime x299 Deluxe II is running an older firmware (3.1.50) that's been reported by others to cause random disconnects and poor transfer speed. I've tried a couple of firmware updaters but they report they're not able to update the adapter even though it is detected and clearly showing that it's running the older firmware.

I've put in a service request to Asus and hopefully they'll be able to provide a working firmware updater. That alone might improve my connection to the 6610, but if not I'll have the Mikrotik CRS305 and SFP+ module to try.

 

[kpfleming]

In the above - would the brocades handle all internal traffic on each Vlan due to the VE interfaces?

Vlan 10 - All 3 switches would route on a layer 3 level?
Vlan 20 - All 3 switches would route on a layer 3 level?
Vlan 30 - Only #1 & #2 would route on a layer 3 level? #3 would not due on Vlan 30 due to no VE 30 interface? #3 would route on layer 3 for Vlan 10 & 20?

Yes, you are slightly thinking about this the wrong direction

In general, except when dynamic routing protocols are in use, each end-device (host) on a LAN will only ever make use of *one* router for reaching other destinations. Routing is an active process, not passive, so the layer 3 switches can't just decide to route some packets and not others, they are asked to route packets.

A VE interface is necessary for routing on that device, but the presence of a VE interface won't (alone) cause any routing to happen. Hosts have to send packets toward that VE interface for routing to happen.

So, in your scenario you should decide which of your two ICX devices you want to handle routing of traffic, and leave the other one configured for just switching. The one that handles routing doesn't have to know about *all* routes in your network, it can route traffic upstream to your other router, but it can certainly handle routing between your VLANs.

If you want every port on both switches to be layer 3 (routing) enabled, you can stack the switches so they become a single logical unit; with that in place, any traffic from a host that needs to be routed to another host on the same physical unit will stay within that unit. Stacking can add a lot of complexity though and upgrades are more difficult, so it's not something you want to do without being ready for it.

 

[RobstarUSA]

Is that the Mikrotik S+RJ10 module? Unfortunately that bumps the price of a solution back into the same range as one of the Supermicro SFP modules that would work with the 6610 @ 5Gbps. Of course the Mikrotik is a lot easier to find than the Supermicro module so perhaps that's what I'll have to do.



Suggestions on a switch that does this without 'breaking the bank'? My understanding is that the CRS305 has 4 SFP+ ports that support 1, 2.5, 5 and 10 Gbps. My current Wiitek SFP module is seen by the 6610 as connecting at 10Gbps, but the system itself reports a 5Gbps link. I suspect I might run into the same issue using it on the CRS305. I purchased the Wiitek as it's affordable, readily available and according the this STH review, it supports 2.5Gbps and 5Gbps NBase-T. It just doesn't have enough buffer to accept the 10Gbps rate that the 6610 thinks it is, even if I rate limit the 6610 SFP port to less than 5Gbps.

https://www.servethehome.com/wiitek-sfp-10g-t-review-10gbase-t-adapter/

I will be using this setup for a few years, but I'm also on a limited budget so I'd like to get the 5Gbps NIC working as affordably as possible. That's why I chose to purchase one of the 6610's, which is working fine for my other 10Gbps and 1Gbps devices.

Yep, IIRC that is the model.

 

[richtj99]

Yes, you are slightly thinking about this the wrong direction

In general, except when dynamic routing protocols are in use, each end-device (host) on a LAN will only ever make use of *one* router for reaching other destinations. Routing is an active process, not passive, so the layer 3 switches can't just decide to route some packets and not others, they are asked to route packets.

A VE interface is necessary for routing on that device, but the presence of a VE interface won't (alone) cause any routing to happen. Hosts have to send packets toward that VE interface for routing to happen.

So, in your scenario you should decide which of your two ICX devices you want to handle routing of traffic, and leave the other one configured for just switching. The one that handles routing doesn't have to know about *all* routes in your network, it can route traffic upstream to your other router, but it can certainly handle routing between your VLANs.

If you want every port on both switches to be layer 3 (routing) enabled, you can stack the switches so they become a single logical unit; with that in place, any traffic from a host that needs to be routed to another host on the same physical unit will stay within that unit. Stacking can add a lot of complexity though and upgrades are more difficult, so it's not something you want to do without being ready for it.

I have my 'main' brocade (#1) connected via lag to #2 & the Main Brocade (#1) is also connected via lag to #3 - so Brocade #1 is the 'main' - first in line of the brocades - so if the routing is on the (#1) I think that would cover #2 & #3 as they are further downstream.

How do I make the routing happen on (#1) - assuming #2 & #3 get their routing from (#1)?

I have multiple Vlans that have no VE interface within the Brocade structure - Vlan 40 & Vlan 45 - if the Brocade has no VE interface for a particular Vlan - does that make it unable to route packets?

As a best practice - should all Brocade devices with Vlan's have VE's?

I did not do VE's on all vlans as I did not want a user on Vlan 40 to be able to access the brocade management console via telnet, ssh, web, etc.

Thanks for the help!

Rich

 

[kpfleming]

I have my 'main' brocade (#1) connected via lag to #2 & the Main Brocade (#1) is also connected via lag to #3 - so Brocade #1 is the 'main' - first in line of the brocades - so if the routing is on the (#1) I think that would cover #2 & #3 as they are further downstream.

How do I make the routing happen on (#1) - assuming #2 & #3 get their routing from (#1)?

I have multiple Vlans that have no VE interface within the Brocade structure - Vlan 40 & Vlan 45 - if the Brocade has no VE interface for a particular Vlan - does that make it unable to route packets?

As a best practice - should all Brocade devices with Vlan's have VE's?

I did not do VE's on all vlans as I did not want a user on Vlan 40 to be able to access the brocade management console via telnet, ssh, web, etc.

Thanks for the help!

Rich

No, that wouldn't be a 'best practice' at all; you don't want to create VEs except in places where you need them. And yes, if there is no VE, then that VLAN is 'layer 2 only' as far as the ICX device is concerned, it cannot do any layer 3 work (routing) in that VLAN.

Your first steps would be to identify the VLANs that you want the ICX #1 to be able to route; in each of those VLANs, create a VE in ICX #1. On ICX #1, add a default route to the 'upstream' router (whatever you are using) so that it can route traffic that is *not* destined for those VLANs to something else which can handle it.

Next step would be to reconfigure at least two hosts to use the VE addresses as their 'default router' or 'default gateway' instead of the upstream router's addresses. With that done, those hosts will send cross-VLAN traffic to ICX #1 for routing, instead of the upstream router. If ICX #1 can route the traffic directly it will, if it cannot it will send the traffic to the upstream router.

As far as restricting access to the management interfaces through the VEs, that can be be done other ways, including access-groups and probably other methods. If *any* IP address on an ICX is reachable from a host, even if it doesn't go through a VE on the same VLAN as the host, then the management interfaces are reachable, so just avoiding creation of a VE in that VLAN won't be sufficient to block that type of access.

 

[NablaSquaredG]

Little rant:

Why on earth did Brocade decide that making standard holes for M6 screws (like literally every other piece of rackmount equipment I have ever had in my hands before) for their rackmount kit is too mainstream and M5 (or probably some weird imperial sh..) is better?

 

[LodeRunner]

I have my 'main' brocade (#1) connected via lag to #2 & the Main Brocade (#1) is also connected via lag to #3 - so Brocade #1 is the 'main' - first in line of the brocades - so if the routing is on the (#1) I think that would cover #2 & #3 as they are further downstream.

How do I make the routing happen on (#1) - assuming #2 & #3 get their routing from (#1)?

I have multiple Vlans that have no VE interface within the Brocade structure - Vlan 40 & Vlan 45 - if the Brocade has no VE interface for a particular Vlan - does that make it unable to route packets?

As a best practice - should all Brocade devices with Vlan's have VE's?

I did not do VE's on all vlans as I did not want a user on Vlan 40 to be able to access the brocade management console via telnet, ssh, web, etc.

Thanks for the help!

Rich

Following kpfleming's post, here's a basic L3 config I mocked up on a 7150:

Spoiler

ICX7150-C12 Router#sh run
Current configuration:
!
ver 08.0.95eT213
!
stack unit 1
  module 1 icx7150-c12-poe-port-management-module
  module 2 icx7150-2-copper-port-2g-module
  module 3 icx7150-2-sfp-plus-port-20g-module
  stack-port 1/3/1
  stack-port 1/3/2
!
global-stp
vlan 1 name DEFAULT-VLAN by port
spanning-tree
!
vlan 11 by port
tagged ethe 1/1/1
untagged ethe 1/1/11
router-interface ve 11
!                                                            
vlan 12 by port
tagged ethe 1/1/1
untagged ethe 1/1/3
router-interface ve 12
!
vlan 20 by port
tagged ethe 1/1/1
router-interface ve 20
!
ip dhcp-client disable
ip route 0.0.0.0/0 172.16.21.2
!

!                                                            
interface ve 11
ip address 10.100.11.1 255.255.255.0
!
interface ve 12
ip address 10.100.12.1 255.255.255.0
ip helper-address 1 10.100.11.2
!
interface ve 20
ip address 172.16.21.1 255.255.255.0
end

In the example, VLAN/VE 20 is the transit to the router; the router needs to have reverse routes on it for each subnet that the switch is handling.

So if Brocade #1 is what connects to your router, you configure every VLAN and VE there and each subnet uses Brocade #1 VE IPs as gateway, along with the default route to the firewall. Your downstream switches just have the required VLANs configured and you trunk them back to #1. The port to your firewall, if physical, would be an untagged port in the transit VLAN.

The ip helper-address statement is because I run a single DHCP server for all pools.

For management on downstream switches, define a VE for the VLAN that normally does management, it will make writing ACLs easier to only worry about one interface in cases where switches don't.

Then as kpfleming said, you would use ACLs to prevent VLAN 40 members from hitting the management interfaces, assuming that VLAN 40 needs routing as well.

 

[deeceesth]

Wanted to hear from anyone else running a similar set up and if they have the same issue as me.

I have a router on a stick set up with a pfsense box and ICX 7250.

I have a port on my ICX 7250 set to untagged VLAN 10, no tagged VLANs, and STP off. This VLAN 10 is used as my WAN connection and the port is directly wired to my cable modem. I then have VLAN 10 tagged on the port that links up to my pfsense LAN port.

pfsense uses VLAN 10 as the WAN interface and it works well. The one issue I run into once in a while (but not 100% reproducible) is when I restart the modem the interface doesn't get a DHCP WAN address. I feel like I have to time the DHCP release/renew in order for pfsense to grab a lease from my ISP.

Essentially I need to start up the modem, wait for the status lights to indicate that its booted up and sees a connection to my ISP and right at that moment I need to do a DHCP renew on my pfsense interface. If i let it sit for longer (let's say 5 mins+) I never get a DHCP lease on that interface even with the manual renew.

Is it possible that the switch is taking the lease? I have DHCP client turned off on the switch and its running L2 FW.

 

[LodeRunner]

Wanted to hear from anyone else running a similar set up and if they have the same issue as me.

I have a router on a stick set up with a pfsense box and ICX 7250.

I have a port on my ICX 7250 set to untagged VLAN 10, no tagged VLANs, and STP off. This VLAN 10 is used as my WAN connection and the port is directly wired to my cable modem. I then have VLAN 10 tagged on the port that links up to my pfsense LAN port.

pfsense uses VLAN 10 as the WAN interface and it works well. The one issue I run into once in a while (but not 100% reproducible) is when I restart the modem the interface doesn't get a DHCP WAN address. I feel like I have to time the DHCP release/renew in order for pfsense to grab a lease from my ISP.

Essentially I need to start up the modem, wait for the status lights to indicate that its booted up and sees a connection to my ISP and right at that moment I need to do a DHCP renew on my pfsense interface. If i let it sit for longer (let's say 5 mins+) I never get a DHCP lease on that interface even with the manual renew.

Is it possible that the switch is taking the lease? I have DHCP client turned off on the switch and its running L2 FW.

I doubt the switch is stealing the IP, but that's easy enough to check by doing "sh ip ad". With L2 firmware your VLANs can't have VE interfaces, and a VE interface can only have a statically assigned IP.

Something I find odd is pfSense losing the IP address. How long does your modem take to restart? Do you get a different public IP each time? Is the modem in true bridge mode or some funky DMZ mode?

 

[deeceesth]

The IP doesn't get lost by pfsense while in use. I was just doing some robustness testing of my setup and noticed when I power cycle my modem I can't get pfsense to get a new lease no matter what I do. I have to do that synchronized power cycle and renew in order for it to work.
The modem is just a straight modem and I get a public IPv4, not funky DMZ or anything. I get a new IP once in a while but not every time. its just standard residential cable.

If I use one of the onboard intel NICs on my pfsense box I don't see any of these issues.

I doubt the switch is stealing the IP, but that's easy enough to check by doing "sh ip ad". With L2 firmware your VLANs can't have VE interfaces, and a VE interface can only have a statically assigned IP.

Something I find odd is pfSense losing the IP address. How long does your modem take to restart? Do you get a different public IP each time? Is the modem in true bridge mode or some funky DMZ mode?

 

[LodeRunner]

After reboot, does traffic still flow to the internet without trying to force a DHCP refresh? Most ISP leases are long enough to cover a CPE restart, so pfSense won't bother refreshing a lease that's more than 50% of the lease time left; it's following RFC.