Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[Craig Curtin]

Does anyone get a moment to look this over and make sure this will work before I send myself down a 2-3 hour rabbit hole again?

OK i had a quick look - i think you havve a basic misunderstanding - yes ports can be dual mode (tagged and untagged traffic) but in your case for the ESXI servers they will be trunk ports - untagged traffic is simply traffic that comes from a device that does not understand VLANs and the switch port it is connected to (or the device such as the Unifi) is in charge of adding the tag - so in other words Tagged and Untagged has nothing to do with the packet on the wire it has to do with what the device at the end will do with it.

If you enable VLANs on the switch all packets that traverse the switch will have VLAN tags on them - some of them might be for the default VLAN for the switch - and when they get to an egress port - the VLAN tag will be stripped.

On a device such as a Unifi - assuming it supports both tagged and untagged clients - then the port the wireless clients connect to will have a PVID - any clients that do not put a tag on their packets (most PCs, phones etc) will have the PVID given to the packet

Craig

 

[LodeRunner]

On a device such as a Unifi - assuming it supports both tagged and untagged clients - then the port the wireless clients connect to will have a PVID - any clients that do not put a tag on their packets (most PCs, phones etc) will have the PVID given to the packet

In UniFi you can assign a VLAN to a SSID and the AP will tag the traffic, so you do a trunk to the AP. For sanity, the trunk port default (untagged) VLAN should be the management VLAN, then tagged VLAN IDs for your client traffic.

 

[SycoPath]

OK i had a quick look - i think you havve a basic misunderstanding - yes ports can be dual mode (tagged and untagged traffic) but in your case for the ESXI servers they will be trunk ports - untagged traffic is simply traffic that comes from a device that does not understand VLANs and the switch port it is connected to (or the device such as the Unifi) is in charge of adding the tag - so in other words Tagged and Untagged has nothing to do with the packet on the wire it has to do with what the device at the end will do with it.

If you enable VLANs on the switch all packets that traverse the switch will have VLAN tags on them - some of them might be for the default VLAN for the switch - and when they get to an egress port - the VLAN tag will be stripped.

On a device such as a Unifi - assuming it supports both tagged and untagged clients - then the port the wireless clients connect to will have a PVID - any clients that do not put a tag on their packets (most PCs, phones etc) will have the PVID given to the packet

Craig

I have absolutely no experience setting up VLANS, so misunderstanding on my end is highly likely. I made a quick and dirty topology in paint for visual aid.

OK, so the switch tags the traffic coming in on a port from a device that doesn't understand vlans. How do I configure a port to pass traffic with VLAN10 and also no vlan (default vlan1)? Everything will still have the vlan bit set to 1 by default, correct?

Unifi allows me to configure an SSID to be on a network tagged with vlan10. So in my case, this traffic will already be arriving at the switch with the vlan10 tag on it, but there will also be traffic on the same switch port (1/1/46 and 1/1/48) with no vlan tags set. I need this traffic to flow, both vlan10 and no vlan, from 1/1/46 (Access Point 1) and 1/1/48 (Access Point 2) to 1/3/1 (trunk1 to ESXi vSwitch) and 1/3/6 (trunk2 to ESXi vSwitch) and to any other clients attached to the ICX6610, assuming they are on the same VLAN.

The whole point of this is network segregation, so I can do some parental controls on my kids network, and also to learn how VLAN's work. I've watched a few YouTube videos, but apparently it's left me with some incorrect assumptions.

According to this link, setting a vSwitch to VLAN 4095 will make it pass all vlan traffic without changing any tags, so if I can get tagged traffic leaving from the ICX6610 then pfSense can handle it from there, and I can have it handle DHCP separately from my default traffic as it should arrive at pfSense tagged VLAN10.

Does my configuration make sense?
What command makes a port into a trunk port on the ICX6610?
If traffic leaves a trunk port headed for pfSense, does it strip the VLAN or pass it still tagged to a vlan?
Do I need to configure any tagging on any ports except 1/1/25 and 1/1/26 for VLAN10? These ports will be desktops with no tag, but the switch should be adding VLAN10 to packets and routing to pfSense

 

[Craig Curtin]

I have absolutely no experience setting up VLANS, so misunderstanding on my end is highly likely. I made a quick and dirty topology in paint for visual aid.

OK, so the switch tags the traffic coming in on a port from a device that doesn't understand vlans. How do I configure a port to pass traffic with VLAN10 and also no vlan (default vlan1)? Everything will still have the vlan bit set to 1 by default, correct?

Unifi allows me to configure an SSID to be on a network tagged with vlan10. So in my case, this traffic will already be arriving at the switch with the vlan10 tag on it, but there will also be traffic on the same switch port (1/1/46 and 1/1/48) with no vlan tags set. I need this traffic to flow, both vlan10 and no vlan, from 1/1/46 (Access Point 1) and 1/1/48 (Access Point 2) to 1/3/1 (trunk1 to ESXi vSwitch) and 1/3/6 (trunk2 to ESXi vSwitch) and to any other clients attached to the ICX6610, assuming they are on the same VLAN.

The whole point of this is network segregation, so I can do some parental controls on my kids network, and also to learn how VLAN's work. I've watched a few YouTube videos, but apparently it's left me with some incorrect assumptions.

According to this link, setting a vSwitch to VLAN 4095 will make it pass all vlan traffic without changing any tags, so if I can get tagged traffic leaving from the ICX6610 then pfSense can handle it from there, and I can have it handle DHCP separately from my default traffic as it should arrive at pfSense tagged VLAN10.

Does my configuration make sense?
What command makes a port into a trunk port on the ICX6610?
If traffic leaves a trunk port headed for pfSense, does it strip the VLAN or pass it still tagged to a vlan?
Do I need to configure any tagging on any ports except 1/1/25 and 1/1/26 for VLAN10? These ports will be desktops with no tag, but the switch should be adding VLAN10 to packets and routing to pfSense

OK so lets take it step by step

A port is a trunk port on a 6610 by adding more than 1 VLAN to it (or in ICX parlance) adding it to more than 1 VLAN

Usually the 4095 VLAN is used for management traffic (by convention - not by hardcoding)

If you have a Unifi connected to a switch port - where is the untagged traffic coming from ?? (ports 1/1/46 and 1/1/48 ?) Turn them into trunk ports by setting all the VLANs they will be connected to. But yes you can make them dual mode and have them also handle untagged traffic - buts makes no sense.

In the case of something like PFsense you want to configure VLANs on PFsense and then attach a single port to your 6610 - this port would be marked as a trunk with all of the VLANs that you would want PFsense to see (if Pfsense is virtual (would not recommend that for a beginner)) then you would assing a single tagged interface through a vswitch connected to the ESXi trunk port

So usually what you would do on your Unifi is setup multiple SSIDs - lets call them Guest, Kids, IOT, Parents - you would assign each of these to a VLAN - lets call them


Guest = VLAN100
Kids=VLAN200
IOT=VLAN300
Parents=VLAN1000

On the switch port that the Unifi connects to you would do the following

conf t
vlan 100 name Guest
exit
vlan 200 name Kids
exit
vlan 300 name IOT
exit
vlan 1000 name Parents
exit

vlan 100
tag e 1/3/1
tag e 1/3/6
tag e 1/1/46

vlan 200
tag e 1/3/1
tag e 1/3/6
tag e 1/1/46
etc etc

This would make the ports 1/3/1 and 1/3/6 and 1/1/46 into trunk ports and able to see and process all traffic for those VLANs.

All of that traffic would be passed on a trunk port to the Pfsense box to be routed/firewalled based on your criteria.

Craig

 

[patg84]

Can a larger than 8gb CF card be installed in the VDX6740?

 

[Craig Curtin]

OK - considering the rear ports are not intended to be used for data ports, you won't find any information on this in the manuals etc. Using them as just data ports is easy enough, but splitting them up and using some for stacking and some for data ports requires some finangling to make the stacking code not try and take over all 4 ports.

However it's possible and stable across reboots etc. basically follow this, it might take you a couple times. This will enable stacking between two units just using the 40gbE qsfp ports, leaving 4x breakout QSFPs total available for data use

#unplug all stacking cables except for 1, going between port 1/2/1 on unit 1 and 1/2/1 on unit two
#don't plug anything else into unit 2 or you'll get a loop
#first you have to unstack all the units and remove any existing stack configuration:
enable
stack unconfigure all
write mem
reload
#both switches should reload, wait till they come back up

#when they're back up, on unit 1:
enable
conf t
stack unit 1
no stack-trunk 1/2/1 to 1/2/2
stack-port 1/2/1 1/2/6
#ignore the warning that pops up about not showing up in "show run"

#still running the below on unit 1!
stack unit 2
no stack-trunk 2/2/1 to 2/2/2
stack-port 2/2/1 2/2/6
#ignore the warning that pops up about not showing up in "show run"
exit
stack enable
write mem
exit
stack secure-setup

#When the setup is done, do "show run"
#stack unit 2 section probably has "stack-trunk 2/2/1 to 2/2/2" added back to it

#the second switch is currently rebooting to rejoin the stack, wait for it to do so
#once it comes back online, do the following to remove the stack unit 2 trunk:
#still on unit 1:
conf t
stack unit 2
no stack-trunk 2/2/1 to 2/2/2
#will probably reboot unit 2 again
#wait for it to come back online, then connect the second pair of 40gbE ports

Once you connect the second pair of 40gbe ports (1/2/6 on unit 1 to 1/2/6 on unit 2) you should be done. Check by running "show stack", you should see both units "ready" with the connection diagram showing both 40gbE ports linked up like below:

ICX1#sh stack
T=6m28.6: alone: standalone, D: dynamic cfg, S: static
ID   Type          Role    Mac Address    Pri State   Comment
1  S ICX6610-48P   active  cc4e.24b8.d9d0 128 local   Ready
2  S ICX6610-48P   standby cc4e.243e.aa74   0 remote  Ready

    active       standby
     +---+        +---+
-2/6| 1 |2/1--2/1| 2 |2/6-
|   +---+        +---+   |
|                        |
|------------------------|
Standby u2 - protocols ready, can failover
Current stack management MAC is cc4e.24b8.d9d0

If you have the same output, do a "write mem" then a "reload" to reload the whole stack. It should come back up fresh in the correct state and running "show stack" again should have the same output as before.

Your final stacking config should look like this:

stack unit 1
  module 1 icx6610-48p-poe-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
  priority 128
  stack-port 1/2/1 1/2/6
stack unit 2
  module 1 icx6610-48p-poe-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
  stack-port 2/2/1 2/2/6
stack enable

@fohdeesha - doing this will allow me to use the other two 40GB slots for straight connection (not fanout) to my Connect X3 cards ? So i could use 2 ports on the first switch for ESXi hosts and one on the 2nd switch for another host ? Or does this config lock the ports into fanout mode etc ?

Craig

 

[fohdeesha]

@fohdeesha - doing this will allow me to use the other two 40GB slots for straight connection (not fanout) to my Connect X3 cards ? So i could use 2 ports on the first switch for ESXi hosts and one on the 2nd switch for another host ? Or does this config lock the ports into fanout mode etc ?

Craig

the breakout ports will always be breakout ports, and the stack connection between switches must always contain 1 full 4gb port. so you're left with 1 40gb port available per switch

 

[fohdeesha]

look mom I made an opnsense juniper

 

[Craig Curtin]

the breakout ports will always be breakout ports, and the stack connection between switches must always contain 1 full 4gb port. so you're left with 1 40gb port available per switch

Aah - OK thats a bummer then - looks like i might leave it as they are and connect up a new Juniper EX4300 i have coming

Craig

 

[guyman]

Hello, was scrolling through this thread a bit and decided to pull the trigger on a 6450-24p (and ended with an extra free 6610-24p). Highly appreciative of all the great information here along with the licencing option. Followed the guide and got my 6450 working beautifully, only concern is power draw. With little to no traffic, one sfp dac to my pc and one sfp 10g tranciever to my router, according to my Kill A Watt I am drawing 50w (around double the estimate) Is this normal?

On another note I ordered Delta EFB0412VHD F00 fans to install, and I am also considering dremeling off the fan grate as mentioned in another post above to quiet the unit even more.

 

[NablaSquaredG]

With little to no traffic, one sfp dac to my pc and one sfp 10g tranciever to my router, according to my Kill A Watt I am drawing 50w (around double the estimate) Is this normal?

25W in the start post doesn't seem to be correct, as I've verified with at least 5 different 6450-24P. They should IDLE around 35W... I only tested with 240V though

 

[guyman]

25W in the start post doesn't seem to be correct, as I've verified with at least 5 different 6450-24P. They should IDLE around 35W... I only tested with 240V though

Hmm, that sucks, oh well. Maybe the new fans will shed a few watts. Wonder if the power supply isnt the most efficient either...

 

[creese]

Hi all,

I recently acquired a pair of ICX 6450-24P's and have a few questions I hope some of you might be able to help with:

1. SSH Connectivity Issue: I've set these up as per the guide, and everything works fine except for SSH access. Whenever I try to SSH into a switch, the connection times out. Surprisingly, I couldn't find any relevant logs about the SSH attempts on the switch. What troubleshooting steps would you recommend?
2. Stacking Configuration: The guide I followed touched on most topics but left stacking as a pending item. Could someone provide insights or resources on how to do this?
3. Network Topology: On a slightly different note, I have a server with dual NICs I'm considering bonding. Is it better to connect these to different switches for redundancy? If I choose this setup, would I still be able to bond the NICs?

Thanks!

 

[kpfleming]

Network Topology: On a slightly different note, I have a server with dual NICs I'm considering bonding. Is it better to connect these to different switches for redundancy? If I choose this setup, would I still be able to bond the NICs?

Yes, if the switches are stacked, they act as a single unit and any ports on them can be included in LAGs. If you want redundancy, then splitting the server's ports across the two physical switches will provide it.

 

[Craig Curtin]

Hi all,

I recently acquired a pair of ICX 6450-24P's and have a few questions I hope some of you might be able to help with:

1. SSH Connectivity Issue: I've set these up as per the guide, and everything works fine except for SSH access. Whenever I try to SSH into a switch, the connection times out. Surprisingly, I couldn't find any relevant logs about the SSH attempts on the switch. What troubleshooting steps would you recommend?
2. Stacking Configuration: The guide I followed touched on most topics but left stacking as a pending item. Could someone provide insights or resources on how to do this?
3. Network Topology: On a slightly different note, I have a server with dual NICs I'm considering bonding. Is it better to connect these to different switches for redundancy? If I choose this setup, would I still be able to bond the NICs?

Thanks!

Are you sure you followed the guide by the OP in this thread - i have done about 6 of the 6450 and not had a single issue with SSH (other than Putty whinging that the cipher is old)

In the same Guide fohdeesha shows how to turn off stacking.

Failing that check the Youtube channel for the Brocade switch config as there is one guide in there on it.

Craig

 

[solloron1]

Hi all,

I have a 7150-c12p and I'm trying to get my guest VLAN ACL sorted. I can make it so the VLAN cannot get out to any other RFC1918 subnet but how can I make it so the main LANs can access them?

Config snippets:

vlan 40 name Guest by port 
 tagged ethe 1/1/11 to 1/1/12 ethe 1/2/1 
 router-interface ve 40 
 spanning-tree 802-1w 
 ip access-group 140 in

ip access-list extended 140
 sequence 10 deny ip 192.168.40.0 0.0.0.255 10.0.0.0 0.255.255.255
 sequence 20 deny ip 192.168.40.0 0.0.0.255 172.16.0.0 0.15.255.255
 sequence 30 deny ip 192.168.40.0 0.0.0.255 192.168.0.0 0.0.255.255
 sequence 40 permit ip 192.168.40.0 0.0.0.255 any

I seem to be missing something. Even if I add permit rule before the deny it allows the guest to access the LAN as well. Thanks

 

[SycoPath]

Huge thanks for the replies. This has been a major help in my understanding of how this all works.

OK so lets take it step by step

A port is a trunk port on a 6610 by adding more than 1 VLAN to it (or in ICX parlance) adding it to more than 1 VLAN

Usually the 4095 VLAN is used for management traffic (by convention - not by hardcoding)

I mentioned vlan 4095 because VMWare states if you set a vSwitch to vlan 4095 it will "pass all traffic unmolested including vlan tags" according to the article I linked, so I assumed this is how to configure a trunk port for a vSwitch since my pfsense is actually virtualized under ESXi. I thought the ESXi vSwitch would drop packets incoming tagged to a vlan it was not configured to be a member of. Is this correct?

If you have a Unifi connected to a switch port - where is the untagged traffic coming from ?? (ports 1/1/46 and 1/1/48 ?) Turn them into trunk ports by setting all the VLANs they will be connected to. But yes you can make them dual mode and have them also handle untagged traffic - buts makes no sense.

UNTAGGED traffic is anything without a vlan tag, correct? Everything from my primary SSID has no vlan tags right now, and is leaving the Unifi AP without vlan tags, making it UNTAGGED traffic, correct? My switch is configured to have the default vlan be VLAN1, so all traffic coming into the switch without vlan tags is considered by the switch, to be VLAN1, correct? Currently all switch ports are members of VLAN1. This also means that when traffic leaves the switch it will be UNTAGGED, correct?

In the case of something like PFsense you want to configure VLANs on PFsense and then attach a single port to your 6610 - this port would be marked as a trunk with all of the VLANs that you would want PFsense to see (if Pfsense is virtual (would not recommend that for a beginner)) then you would assing a single tagged interface through a vswitch connected to the ESXi trunk port

So usually what you would do on your Unifi is setup multiple SSIDs - lets call them Guest, Kids, IOT, Parents - you would assign each of these to a VLAN - lets call them


Guest = VLAN100
Kids=VLAN200
IOT=VLAN300
Parents=VLAN1000

I plan on adding more VLANs, each with it's own DHCP and /24 IP space restricting inter-vlan communication by utilizing pfsense firewall rules later. Once I wrap my head around setting up a single vlan and getting it working, I'll be adding more. I figured getting a single VLAN working correctly would be a lot easier than trying to get 5-6 working all at once.

On the switch port that the Unifi connects to you would do the following

conf t
vlan 100 name Guest
exit
vlan 200 name Kids
exit
vlan 300 name IOT
exit
vlan 1000 name Parents
exit

vlan 100
tag e 1/3/1
tag e 1/3/6
tag e 1/1/46

vlan 200
tag e 1/3/1
tag e 1/3/6
tag e 1/1/46
etc etc

This would make the ports 1/3/1 and 1/3/6 and 1/1/46 into trunk ports and able to see and process all traffic for those VLANs.

All of that traffic would be passed on a trunk port to the Pfsense box to be routed/firewalled based on your criteria.

Craig

This helps a lot and is definitely what I was asking someone to share with me. Adding a port to be a member of multiple vlans will let traffic from ANY of the configured member vlans egress out of that port. In the configuration above, all packets leaving 1/3/1 will retain it's VLAN tag, correct? So from the above example, packets would egress 1/3/1 with VLAN100, VLAN200, VLAN300, and VLAN1000 tags intact headed for ESXi's vSwitch, correct? Also, setting the dual-mode for ports would allow all member VLAN traffic to flow, as well as UNTAGGED traffic, correct?

I currently have pfSense connected to the ESXi port group 4095, as this is how I thought trunk ports are set up for vSwitches, and this seems to be working as expected, since I still have an internet connection (See attached image).

 

[tr_deal]

First I want to give a huge shout out to fohdeesha! The guides and this thread we're the reasons I picked up a 6610 and integrated it into my homelab.

I have been banging my head on one thing and was wondering if anyone had any suggestions. I have 4 Amcrest PoE cameras on vLAN50. No matter what I do the switch only negotiates 100full to the cameras. If I try to force 1000master I lose connection.

I've upgraded the firmware on the cameras to the latest and verified when I move them to my Aruba 2930m that switch negotiates them to 1000full. The only thing that keeps sticking out to me is when I look at the Inline Power Statistics for the camera ports it shows the correct type (802.3af) but for class it says n/a. On the ports where my WAPs are connected is has both the correct type and class (Class 3). This may be nothing but its the only difference I see.

Any thoughts or pointers would be appreciated!

 

[LodeRunner]

First I want to give a huge shout out to fohdeesha! The guides and this thread we're the reasons I picked up a 6610 and integrated it into my homelab.

I have been banging my head on one thing and was wondering if anyone had any suggestions. I have 4 Amcrest PoE cameras on vLAN50. No matter what I do the switch only negotiates 100full to the cameras. If I try to force 1000master I lose connection.

I've upgraded the firmware on the cameras to the latest and verified when I move them to my Aruba 2930m that switch negotiates them to 1000full. The only thing that keeps sticking out to me is when I look at the Inline Power Statistics for the camera ports it shows the correct type (802.3af) but for class it says n/a. On the ports where my WAPs are connected is has both the correct type and class (Class 3). This may be nothing but its the only difference I see.

Any thoughts or pointers would be appreciated!

Try just '1000-full' instead of '1000-full-master'? Or is just '1000-full' not available on the 6610?

 

[bwahaha]

it shows the correct type (802.3af) but for class it says n/a

I'm pretty stoned right now, but I can't seem to find any way to set the class on a port via cli. Ironically, it's right there in the web gui. Might try setting those ports to an appropriate class there.

chart for easyish reference:

Power over Ethernet - Wikipedia

en.wikipedia.org