Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[TonyArrr]

Having never done this before its easy to mess up, I think half the battle is everyone is excited to play with their new switch and dont freaking read the guide.

The excitement worked in my favour actually. I read the guide repeatedly while my switch was shipping down under cause it was taking too damn long

 

[NablaSquaredG]

Oh god... our friend masscloseouts2000 has done it again.

What I ordered: 64x 8Gb LW (SFP+ Singlemode) -> $1 per piece
What I got: 64x 4x16Gb SW (QSFP+ Multimode MTP) -> 7$ per piece - also known as the glorious Brocade XBR-000232 or 57-1000267-01

Very expensive misdelivery for them and also for me, because it's not what I wanted.

If anyone in Germany / Europe needs some cheap XBR-000232 or 57-1000267-01 for cheap 40G connectivity, let me know (guide is here https://forums.servethehome.com/ind...gbe-40gbe-switching.21107/page-23#post-201367)

 

[Sana128]

Hello everyone,
right now I have a Mikrotik hAP ac2 router configured as a Router on a Stick with a Cisco WS-C3750G-24PS-S switch. I have a few VLANS (I can provide the router config) for my home lab.

1) I recently bought a Layer 3 switch ICX6610 and hope to do the VLAN routing on the Switch level (I already Unlocked and created VLANS on the switch with the same VLAN IDs thanks to @fohdeesha ). What is the best way to achieve this?

Edit: I already set up the Inter VLAN routing as described on this youtube video. but I am not sure if that's enough?

2) I might have to upgrade the router as it only supports gigabit internet, what would be the best bang for the buck router (since I already have an L3 switch), I am open to trying non-Makrotik routers as well.
3) Do you think I can get away with ISP ISP-provided router since the switch has ACL? so I don't need a fancy router?

 

[itronin]

Hello everyone,
right now I have a Mikrotik hAP ac2 router configured as a Router on a Stick with a Cisco WS-C3750G-24PS-S switch. I have a few VLANS (I can provide the router config) for my home lab.

1) I recently bought a Layer 3 switch ICX6610 and hope to do the VLAN routing on the Switch level (I already Unlocked and created VLANS on the switch with the same VLAN IDs thanks to @fohdeesha ). What is the best way to achieve this?

Edit: I already set up the Inter VLAN routing as described on this youtube video. but I am not sure if that's enough?

Congrats!
Enough for?

Mikrotik is a Firewall & Router (like most home "routers"). While it may be light on ACL's due to the processing power (IDK - never used one), it provides an all important NAT function. Whether the Mikrotik is a limiter will also depend on the speed of your Internet connection and how many ACL's (assuming it has those) it can have and process and what you want to limit.

2) I might have to upgrade the router as it only supports gigabit internet, what would be the best bang for the buck router (since I already have an L3 switch), I am open to trying non-Makrotik routers as well.

You have a great Router, its called an ICX6610.
NB: Firewall != Router

3) Do you think I can get away with ISP ISP-provided router since the switch has ACL? so I don't need a fancy router?

Probably not. At a guess you probably cannot get enough public IP's from your provider so you'll be using private IP's on your internal network.
You need something to provide NAT - Network Address Translation.

There are a few different ways to design Internet egress with what you have, depends on requirements and need I suppose.
Try searching here for transit vlan, Internet transit vlan etc.

If you have spare PC hardware you may also want to look at one of the router OS distributions: opnsense, pfsense, vyos, dd-wrt, etc. etc. etc.

you are about to fall down the rabbit hole!

 

[Sana128]

There are a few different ways to design Internet egress with what you have, depends on requirements and need I suppose.
Try searching here for transit vlan, Internet transit vlan etc.

If you have spare PC hardware you may also want to look at one of the router OS distributions: opnsense, pfsense, vyos, dd-wrt, etc. etc. etc.

you are about to fall down the rabbit hole!

I already have a transit VLAN setup between an ICX6610-48P Layer 3 Switch and a MikroTik HAP ac² router with the same VLANs.

Step 1: Create VLANs

1. Configure VLANs on the ICX6610-48P Layer 3 Switch: I have already configured VLANs on the ICX6610-48P switch. I have assigned IP addresses to the VLAN interfaces (Ve) on the switch.
   
   
2. Connect the ICX6610-48P Switch and MikroTik Router: Connected one of the switch's ports (ether4) to a port on the MikroTik router. This will be the transit link for inter-VLAN routing.
   
   
3. Configure Routing on the MikroTik Router: On the MikroTik router, I configured routing to allow traffic between VLANs.

/ip route
add dst-address=10.0.10.0/24 gateway=10.0.10.1
add dst-address=10.0.15.0/24 gateway=10.0.15.1
add dst-address=10.0.20.0/24 gateway=10.0.20.1
add dst-address=10.0.30.0/24 gateway=10.0.30.1
add dst-address=10.0.40.0/24 gateway=10.0.40.1
add dst-address=10.0.50.0/24 gateway=10.0.50.1
add dst-address=10.0.99.0/24 gateway=10.0.99.1

Step 2: Create a Transit VLAN

(config)$ int ve 50               #VE 50 configuration
(config-vif-50)$ ip address 10.0.50.1/24  #Assign an IP address to VE 50
(config-vif-50)$ exit

Step 3: Configure the MikroTik Router

Configure an IP address on the router's Ethernet port (ether4) in the same transit network:
/ip address add address=10.0.50.2/24 interface=ether4

I was able to ping and verify the connectivity.

I don't know what else I have to do at this point, not sure how things like DHCP should be assigned (should I disable that on the router?)

 

[itronin]

...
I was able to ping and verify the connectivity.

I don't know what else I have to do at this point, not sure how things like DHCP should be assigned (should I disable that on the router?)

so if it works then your question is?

Now let me ask you somethings because I am curious.

Are all of the ".1" addresses ve interfaces on your 6610?

and your MikroTik has its Internet connection, the transit vlan at address 10.0.50.2/24 and DOES NOT have any interfaces (VLAN or PHY) in any of the VLAN's you're adding routes for (.10,.15,.20.30.40.50.99)?

When you say ping and verify - was that from the MikroTik? If it was and your MiktroTik only has Internet and Transit then you may not be pinging your ICX VLAN interfaces (except for 50.1).

by the way you should NOT have to add to your MIkroTik:

add dst-address=10.0.50.0/24 gateway=10.0.50.1

If your MikroTik's 50.2 interface is propertly defined as a /24. that route should already exist as a "direct" or directly connected route.

I'd expect all those MikroTik routes you added to have the same gateway which is the ICX's ve IP in your transit VLAN - hence my questions about whether your MikroTik has other vlan interfaces defined besides your transit (and your Internet facing connection).

you may want to understand what I am asking above before moving on to DHCP

For DHCP you will set a helper-address in for the vlans you want another device (MikroTik?) to manage DHCP for those VLANs.
TH has a video on this topic.

However doing a quick search I saw something that basically said (as of 2021)"AFAIK MikroTik routerOS DHCP server does not support relay information." But maybe that functionality has been added since that post *or* the responder to the OP there was incorrect. you'll need to research that. NB: neither opn/pf sense firewalls support it (though rumor has it that opnsense is looking at a better dhcp server that will support it).
IDK about whether dhcp server on pihole will or will not and same for vyos as I haven't played with those.

if you want a nice gui dhcp server that does support relay and you have a windows server then use that as your helpder destination.

You DO have a dhcp server in the ICX however folks more learned than me have essentially said "don't use it - its broken". so YMMV if you do go down that path.

 

[Sana128]

so if it works then your question is?

Now let me ask you somethings because I am curious.

Are all of the ".1" addresses ve interfaces on your 6610?

previously they were on Mikrotik but I recreated them on 6610

and your MikroTik has its Internet connection, the transit vlan at address 10.0.50.2/24 and DOES NOT have any interfaces (VLAN or PHY) in any of the VLAN's you're adding routes for (.10,.15,.20.30.40.50.99)?

Previously that was not the case but since then I created a new VLAN that is exclusive to 6610 (please see below)

When you say ping and verify - was that from the MikroTik? If it was and your MiktroTik only has Internet and Transit then you may not be pinging your ICX VLAN interfaces (except for 50.1).

I'd expect all those MikroTik routes you added to have the same gateway which is the ICX's ve IP in your transit VLAN - hence my questions about whether your MikroTik has other vlan interfaces defined besides your transit (and your Internet facing connection).

I was able to ping the switch from the router side and the router from the switch
There are other VLANs on that router I am not able to disable all without interrupting others at the house. I am planing to replace them with ICX 6610 but that needs to be up and running before I disconnect the router VLANs.

For DHCP you will set a helper-address in for the vlans you want another device (MikroTik?) to manage DHCP for those VLANs.
TH has a video on this topic.

SSH@ICX6610(config)#vlan 17
SSH@ICX6610(config-vlan-17)#untagged e 1/1/17
Added untagged port(s) ethe 1/1/17 to port-vlan 17.
SSH@ICX6610(config-vlan-17)#router-interface ve 17
SSH@ICX6610(config-vlan-17)#exit
SSH@ICX6610(config)#int ve 17
SSH@ICX6610(config-vif-17)#ip address 10.0.17.1/24
SSH@ICX6610(config-vif-17)#show int ve 17


## Mikrotik

/ip pool
add name=17_POOL ranges=10.0.17.2-10.0.17.254

/ip dhcp-server network
add address=10.0.17.0/24 dns-server=8.8.8.8 gateway=10.0.17.1

/ip dhcp-server

add address-pool=17_POOL interface=Transit name=17 relay=10.0.17.1

/ip route

add disabled=no dst-address=10.0.17.0/24 gateway=Transit routing-table=main \
    suppress-hw-offload=no


## On 6610

Int ve 17
Ip helper-address 1 10.0.50.2
exit

With that, I am successfully getting a DHCP address and able to ping other VLANs and the router But there is no internet. I suspect that had something to do with the firewall rules on the router. I just need to confirm that I’m on the right track. Networking isn’t my forte, which is why I’ve taken up this hobby.

Edit : I was able to fix the firewall issue and now the switch can ping the internet

SSH@ICX6610#ping 8.8.8.8
Sending 1, 16-byte ICMP Echo to 8.8.8.8, timeout 5000 msec, TTL 64
Type Control-c to abort
Reply from 8.8.8.8         : bytes=16 time=24ms TTL=60
Success rate is 100 percent (1/1), round-trip min/avg/max=24/24/24 ms.

However, a PC connected to the VLAN 17 does not have internet and cannot ping the router.

 

[shpitz461]

Ok, I got the ECS2512FP and connected it to the 6610 using two 10GbE fiber connections.
I also connected a WAP to the 2.5GbE PoE port of the ECS2512FP.
Where do I go from here? I think I need to set the two SFP+ ports to aggregate.
My networking knowledge is very basic, I don't know what the hell I'm doing...
I tried following Terry Henry's videos but he's not starting from the point of a beginner.
I don't have any VLAN's or anything set up, just simple single-layout network which I know is a mess.
I'm just trying to be a better boy...
Can someone point me to some good videos on home-network setup 101?
Thanks!

EDIT:
I have the following:
AT&T fiber connected to an EdgeRouter Lite (erlite-3).
Router connected to the 6610
The 6610 is the focal point, everything is connected to it.
IoT
HDHomerun tuners
WAP (now via the ECS2512FP to the 6610 via two SFP+ fiber connections)
VOIP
Printers
NAS
Laptops (all wireless)
Desktops
Doorbell camera
IP Cameras

 

[sic0048]

Ok, I got the ECS2512FP and connected it to the 6610 using two 10GbE fiber connections.
I also connected a WAP to the 2.5GbE PoE port of the ECS2512FP.
Where do I go from here? I think I need to set the two SFP+ ports to aggregate.
My networking knowledge is very basic, I don't know what the hell I'm doing...
I tried following Terry Henry's videos but he's not starting from the point of a beginner.
I don't have any VLAN's or anything set up, just simple single-layout network which I know is a mess.
I'm just trying to be a better boy...
Can someone point me to some good videos on home-network setup 101?
Thanks!

EDIT:
I have the following:
AT&T fiber connected to an EdgeRouter Lite (erlite-3).
Router connected to the 6610
The 6610 is the focal point, everything is connected to it.
IoT
HDHomerun tuners
WAP (now via the ECS2512FP to the 6610 via two SFP+ fiber connections)
VOIP
Printers
NAS
Laptops (all wireless)
Desktops
Doorbell camera
IP Cameras

I would say the easiest thing for a beginner to do is to run the switches as L2 devices and put all your rules, VLANs, etc in the firewall/router device you are running. This way there is very little that you need to set up in the switch other than perhaps some VLAN tagging on ports (if needed). Otherwise it will pretty much be plug and play while your firewall/router does all the heavy lifting.

How to set up your firewall/router depends heavily on the device/software you are using. There are plenty of YouTube "how to" videos for the firewall software pfSense if you are using that solution.

These switches are obviously designed to run as L3 devices (where you have the switch handle routing instead of the router/firewall device). This is theoretically a quicker way to move data, but likely an unnecessary step in a typical home network. What I mean is that a typical home setting isn't moving such high levels of data that you'll actually see a benefit from this type of setup. Setting up the switches as L3 devices is also going to be much more difficult for a someone that "doesn't know what the hell they are going".

 

[JacobRhea]

show conf
!
Startup-config data location is flash memory
!
Startup configuration:
!
ver 08.0.30tT7f3
!
stack unit 1
  module 1 icx6610-48p-poe-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
!
!
!
!
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
!
vlan 10 name SERVERS by port
untagged ethe 1/1/25 to 1/1/36 ethe 1/2/6 ethe 1/3/1 to 1/3/4
router-interface ve 10
!
vlan 11 name PRODUCTION by port
!
vlan 12 name STAGING by port
!
vlan 20 name SECURE by port
tagged ethe 1/1/48
untagged ethe 1/1/1
router-interface ve 20
!
vlan 99 name MGMT by port
untagged ethe 1/1/13 to 1/1/24
router-interface ve 99
!
vlan 100 name GUEST by port
tagged ethe 1/1/48
router-interface ve 100
!
vlan 101 name IOT by port
tagged ethe 1/1/48
router-interface ve 101
!
!
!
!
!
aaa authentication web-server default local
aaa authentication enable default local
aaa authentication login default local
enable telnet authentication
enable aaa console
hostname switch
ip dhcp-client disable
ip dns server-address 10.0.0.1
ip route 0.0.0.0/0 10.0.0.1
ip add-host-route-first
!
no telnet server
username username password .....
!
!
clock summer-time
clock timezone gmt GMT-05
!
!
ntp
disable serve
server 216.239.35.0
server 216.239.35.4
!
!
!
!
!
!
!
!
!
interface ethernet 1/1/48
dual-mode
inline power
!
interface ve 1
ip address 10.0.0.2 255.255.255.248
ip helper-address 1 10.0.0.1
!
interface ve 10
ip address 10.0.10.1 255.255.255.0
ip helper-address 1 10.0.0.1
!
interface ve 20
ip address 10.0.20.1 255.255.255.0
ip helper-address 1 10.0.0.1
!
interface ve 99
ip address 10.0.99.1 255.255.255.0
ip helper-address 1 10.0.0.1
!
interface ve 100
ip address 10.0.100.1 255.255.255.0
ip helper-address 1 10.0.0.1
!
interface ve 101
ip address 10.0.101.1 255.255.255.0
ip helper-address 1 10.0.0.1
!
!
!
!
!
!
!
!
!
end

I'm trying to setup my Proxmox servers on VLAN10 and have the VM's running on them on VLAN11/12 but I'm having issues getting the setup right, I've tried a variety of tagged/untagged setups and haven't had any success. Can someone point me in the right direction?

 

[bwahaha]

show conf
!
Startup-config data location is flash memory
!
Startup configuration:
!

I know bugger all about vlans, so take this with a grain of salt... Try not doing the dual-mode, if you don't want default/vlan1 on the port?

 

[JacobRhea]

Made some changes and got it all working. I didn't quite understand how VLANs worked and what tagged vs untagged meant. Watched a couple videos and read up now it's all working great!

 

[fohdeesha]

Made some changes and got it all working. I didn't quite understand how VLANs worked and what tagged vs untagged meant. Watched a couple videos and read up now it's all working great!

this is the way

 

[i386]

this is the way

 

[shpitz461]

Made some changes and got it all working. I didn't quite understand how VLANs worked and what tagged vs untagged meant. Watched a couple videos and read up now it's all working great!

Can you post your working config?

 

[Tekgnome]

Is anyone using opnsense with a 6610

I cannot for the life of me get it to work.

From a client PC I can ping the firewall - no drop packets
From the firewall I can't ping the client - 3/3 dropped packets
From the firewall I can ping the switch but I only get 1/3 packets.

My WAN and LAN have the correct Ethernet port assigned

I'm running 192.168.1.1/24 and I also configured a my interface as vlan 1

Edit:
It's now intermittently working. I can get Internet for a few seconds on certain clients and then it drops off again.

Client PCs see the DHCP server on 192.168.1.1

 

[hmw]

Is anyone using opnsense with a 6610

I cannot for the life of me get it to work.

From a client PC I can ping the firewall - no drop packets
From the firewall I can't ping the client - 3/3 dropped packets
From the firewall I can ping the switch but I only get 1/3 packets.

My WAN and LAN have the correct Ethernet port assigned

I'm running 192.168.1.1/24 and I also configured a my interface as vlan 1

Edit:
It's now intermittently working. I can get Internet for a few seconds on certain clients and then it drops off again.

Client PCs see the DHCP server on 192.168.1.1

Are you running your 6610 as a switch or as a router?

 

[Tekgnome]

Are you running your 6610 as a switch or as a router?

I believe just a switch.

I followed the guide on setup and it seems to work just fine plugged into my att gateway

 

[hmw]

I believe just a switch.

I followed the guide on setup and it seems to work just fine plugged into my att gateway

I’ve used the 6610 for a few years with OPNsense. Now using a 7650. Never had problems, although the 7650 is now running as the core router and OPNsense is purely a gateway firewall. Drawing your network would make it easier to figure out what’s going on

 

[EngineerNate]

Hi all, I've got a weird one with my 7450-48P. Updated to the latest firmwares on Fohdeesha's site.

It's powering an ubiquiti unifi 6 enterprise happily right up until I plug a desktop (not Poe) into a neighboring port. Then no PoE on any port works. I could watch the light go out on the AP as soon as I plugged the desktop in and start booting back up as soon as I removed it.

The same desktop/cable don't seem to be causing issues on my Nortel switch the Brocade is supposed to be replacing.

Any ideas?

Cheers,
Nate