Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[SycoPath]

Looking at what you pasted, I don't see that you've added any ports to the VLAN as either tagged or untagged. I only have 7k series units, so my syntax is slightly different for trunking, but for a pure trunk port where you have may not need a default/native/untagged VLAN, the command should be similar to:

SSH@office-sw(config-vlan-4)#tag e 1/3/1

Issued inside the VLAN context.

Edge ports to workstations would be untagged, like:

SSH@office-sw(config-vlan-4)#untag e 1/1/1

I'm still missing it I think, sorry. Adding switch ports to a vlan tells the switch which vlan that port should be, but you also need to issue the tag e port# as well so tagged/untagged traffic will flow? This might be what I was missing.

For example for my config, I have two workstations that should be vlan 10, 1/1/23 and 1/1/24. Everything else should be untagged on the default vlan1, except trunk ports. Trunk ports are 1/3/1 and 1/3/6 for the connection from esxi to the icx6610, and both of these should be dual mode, correct? Same with 1/1/46 and 1/1/48. These come from the unifi AP's and will be carrying both vlan10 and vlan1 traffic, so they need to be dual mode and tagged as well, correct?

I need tagged and untagged to flow on trunk ports, and I'm not understanding how to do this.

Does this look correct?

Vlan 1

SSH@ICX6610-48P(config-vlan-1)# Untagged eth 1/1/1 to 1/1/22
SSH@ICX6610-48P(config-vlan-1)# Untagged eth 1/1/25 to 1/1/45
SSH@ICX6610-48P(config-vlan-1)# Untagged eth 1/1/46 to 1/1/48
SSH@ICX6610-48P(config-vlan-1)# Untagged eth 1/3/1 
SSH@ICX6610-48P(config-vlan-1)# Untagged eth 1/3/6

Vlan 10

ICX6610-48P(config-vlan-10)# Untagged eth 1/1/24 to 1/1/25
ICX6610-48P(config-vlan-10)# Tagged eth 1/1/46 to 1/1/48
ICX6610-48P(config-vlan-10)# Tagged eth 1/3/1 
ICX6610-48P(config-vlan-10)# Tagged eth 1/3/6

Dual mode for trunk ports

SSH@ICX6610-48P(config)# int eth 1/1/46
SSH@ICX6610-48P(config-if-e10000-1/1/46)# dual-mode
SSH@ICX6610-48P(config)# int eth 1/1/47
SSH@ICX6610-48P(config-if-e10000-1/1/47)# dual-mode
SSH@ICX6610-48P(config)# int eth 1/1/48
SSH@ICX6610-48P(config-if-e10000-1/1/48)# dual-mode
SSH@ICX6610-48P(config)# int eth 1/3/1
SSH@ICX6610-48P(config-if-e10000-1/3/1)# dual-mode
SSH@ICX6610-48P(config)# int eth 1/3/6
SSH@ICX6610-48P(config-if-e10000-1/3/6)# dual-mode

 

[Ralph_IT]

Hi,
I bought an 7250-48P but I cannot see the console nor I am able to find a way to enter its configuration or http page.
I'm using this mini-usb to RS232 cable (don't know if it is valid) but after opening the COM port with putty, all I see is a black screen. No messages, no boot, no login...
I think the switch does boot up and loads the OS, as the fans begin at full speed and a couple of minutes later they rev down.

Any ideas to where to go from here?

 

[kpfleming]

It's hard to say whether that cable will work or not, since the listing you linked does not show how the connectors are wired.

 

[Ralph_IT]

It's hard to say whether that cable will work or not, since the listing you linked does not show how the connectors are wired.

Thanks for the quick reply.
I've searched for the pinout but it seems it's nowhere to be found.
Found a DIY tuto on Reddit which implies shredding a micro usb and a Cisco cable.

 

[LodeRunner]

Thanks for the quick reply.
I've searched for the pinout but it seems it's nowhere to be found.
Found a DIY tuto on Reddit which implies shredding a micro usb and a Cisco cable.

Yes, that's pretty much it. buy the official Ruckus cable or hack one together. If a previous owner plugged in a standard USB cable expecting a USB to serial UART on the far end, then they most likely burned it out completely since they would have hit the serial port with USB line voltage.

Maybe this is the Reddit post you found: Reddit post

I don't know if that was the post I followed, but I chopped up a USB cable and attached it to a keystone so I didn't have to mangle my perfectly good Cisco cable. I have since bought a proper Ruckus cable.

 

[kpfleming]

The official cable/adapter combos are available if you look for them; some of us have spares and are willing to sell them too, if that's something you want to consider.

 

[Ralph_IT]

Yes, that's pretty much it. buy the official Ruckus cable or hack one together. If a previous owner plugged in a standard USB cable expecting a USB to serial UART on the far end, then they most likely burned it out completely since they would have hit the serial port with USB line voltage.

Maybe this is the Reddit post you found: Reddit post

I don't know if that was the post I followed, but I chopped up a USB cable and attached it to a keystone so I didn't have to mangle my perfectly good Cisco cable. I have since bought a proper Ruckus cable.

That's one of the 2 post on Reddit that I found (Here's the other).
What are the chances of configuring a switch with a burnt usb serial port?

The official cable/adapter combos are available if you look for them; some of us have spares and are willing to sell them too, if that's something you want to consider.

I wouldn't mind buying one official cable if I can make the switch work first.

 

[bwahaha]

What are the chances of configuring a switch with a burnt usb serial port?

how are you with component repairs? Most likely, it'll have popped the uart chip. They may have thought ahead and put some sacrificial components in the path that can be swapped out.

It also looks like it may be a daughter board.

https://forums.servethehome.com/index.php?threads/brocade-icx-series-cheap-powerful-10gbe-40gbe-switching.21107/post-299346

 

[fohdeesha]

That's one of the 2 post on Reddit that I found (Here's the other).
What are the chances of configuring a switch with a burnt usb serial port?


I wouldn't mind buying one official cable if I can make the switch work first.

I would bet $50 the console port is fine, tons of people just in this thread have accidentally plugged regular USB ports into them, they can handle it fine. your issue is probably your usb to serial adapter, use it with another serial device to verify it actually works, or buy an ftdi model off amazon. also official console cables for these are only like 20 bucks to take the guesswork out of wiring Brocade ICX7250, ICX7450 ICX7750 DB9 Female to USB Mini Serial Console Cable 6FT | eBay

 

[SycoPath]

I'm still missing it I think, sorry. Adding switch ports to a vlan tells the switch which vlan that port should be, but you also need to issue the tag e port# as well so tagged/untagged traffic will flow? This might be what I was missing.

For example for my config, I have two workstations that should be vlan 10, 1/1/23 and 1/1/24. Everything else should be untagged on the default vlan1, except trunk ports. Trunk ports are 1/3/1 and 1/3/6 for the connection from esxi to the icx6610, and both of these should be dual mode, correct? Same with 1/1/46 and 1/1/48. These come from the unifi AP's and will be carrying both vlan10 and vlan1 traffic, so they need to be dual mode and tagged as well, correct?

I need tagged and untagged to flow on trunk ports, and I'm not understanding how to do this.

Does this look correct?

Vlan 1

SSH@ICX6610-48P(config-vlan-1)# Untagged eth 1/1/1 to 1/1/22
SSH@ICX6610-48P(config-vlan-1)# Untagged eth 1/1/25 to 1/1/45
SSH@ICX6610-48P(config-vlan-1)# Untagged eth 1/1/46 to 1/1/48
SSH@ICX6610-48P(config-vlan-1)# Untagged eth 1/3/1
SSH@ICX6610-48P(config-vlan-1)# Untagged eth 1/3/6

Vlan 10

ICX6610-48P(config-vlan-10)# Untagged eth 1/1/24 to 1/1/25
ICX6610-48P(config-vlan-10)# Tagged eth 1/1/46 to 1/1/48
ICX6610-48P(config-vlan-10)# Tagged eth 1/3/1
ICX6610-48P(config-vlan-10)# Tagged eth 1/3/6

Dual mode for trunk ports

SSH@ICX6610-48P(config)# int eth 1/1/46
SSH@ICX6610-48P(config-if-e10000-1/1/46)# dual-mode
SSH@ICX6610-48P(config)# int eth 1/1/47
SSH@ICX6610-48P(config-if-e10000-1/1/47)# dual-mode
SSH@ICX6610-48P(config)# int eth 1/1/48
SSH@ICX6610-48P(config-if-e10000-1/1/48)# dual-mode
SSH@ICX6610-48P(config)# int eth 1/3/1
SSH@ICX6610-48P(config-if-e10000-1/3/1)# dual-mode
SSH@ICX6610-48P(config)# int eth 1/3/6
SSH@ICX6610-48P(config-if-e10000-1/3/6)# dual-mode

Does anyone get a moment to look this over and make sure this will work before I send myself down a 2-3 hour rabbit hole again?

 

[Ralph_IT]

how are you with component repairs? Most likely, it'll have popped the uart chip. They may have thought ahead and put some sacrificial components in the path that can be swapped out.

It also looks like it may be a daughter board.

https://forums.servethehome.com/index.php?threads/brocade-icx-series-cheap-powerful-10gbe-40gbe-switching.21107/post-299346

I would bet $50 the console port is fine, tons of people just in this thread have accidentally plugged regular USB ports into them, they can handle it fine. your issue is probably your usb to serial adapter, use it with another serial device to verify it actually works, or buy an ftdi model off amazon. also official console cables for these are only like 20 bucks to take the guesswork out of wiring Brocade ICX7250, ICX7450 ICX7750 DB9 Female to USB Mini Serial Console Cable 6FT | eBay

Luckyly for me, @fohdeesha was right and culprit was the mini USB adapter I was using, because I suck at manual skills requiring precision.
I did a guetto mini usb to serial adapter following the reddit guide and it worked, but needed to redo the wiring more than 10 times.

I'm following the guide right now to have the latest firmware flashed.
Time to check if all ports work normally, POE included.

Definitely I want a proper console cable, so will post a WTB on the forum soon.

 

[Craig Curtin]

Hey Guys,

I have a couple of 6610s here - one is my testing and messing around one and the other in production.

Currently i am using two of the rear ports at 40Gb/s into Mellanox cards in my ESXi cluster.

I could really use a 3rd one though for my final host

At the moment i am using the 2 x extra ports (stacking) at the back with a fanout 4x10Gb cable to other hosts

Is it possible to turn stacking on for just one of the ports on the rear so i could stack a 2nd 6610 and then use its 2 x 40Gb ports to connect my hosts and use one fanout on each of the switches to connect other clients ?

So it would look like

2 x 40GB on the first switch to ESXI hosts
1 x Fanout from stacking port on first switch to 10GB clients
1 x stacking port on first switch at 40Gb to 2nd switch

1 x 40GB port on 2nd switch to ESXi host ?

So the basic question i guess is once the stacking is turned on - are both ports put into stacking mode ?

Do i just use a stadnard QSFP cable to stack both switches ?

Craig

 

[MrGuvernment]

That's one of the 2 post on Reddit that I found (Here's the other).
What are the chances of configuring a switch with a burnt usb serial port?


I wouldn't mind buying one official cable if I can make the switch work first.

I will confirm, this cable works fine if you need to buy one, i own it and use it often with my 6450

https://www.amazon.ca/gp/product/B076Z8RN5Q/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1

 

[Craig Curtin]

Does anyone get a moment to look this over and make sure this will work before I send myself down a 2-3 hour rabbit hole again?

OK i had a quick look - i think you havve a basic misunderstanding - yes ports can be dual mode (tagged and untagged traffic) but in your case for the ESXI servers they will be trunk ports - untagged traffic is simply traffic that comes from a device that does not understand VLANs and the switch port it is connected to (or the device such as the Unifi) is in charge of adding the tag - so in other words Tagged and Untagged has nothing to do with the packet on the wire it has to do with what the device at the end will do with it.

If you enable VLANs on the switch all packets that traverse the switch will have VLAN tags on them - some of them might be for the default VLAN for the switch - and when they get to an egress port - the VLAN tag will be stripped.

On a device such as a Unifi - assuming it supports both tagged and untagged clients - then the port the wireless clients connect to will have a PVID - any clients that do not put a tag on their packets (most PCs, phones etc) will have the PVID given to the packet

Craig

 

[LodeRunner]

On a device such as a Unifi - assuming it supports both tagged and untagged clients - then the port the wireless clients connect to will have a PVID - any clients that do not put a tag on their packets (most PCs, phones etc) will have the PVID given to the packet

In UniFi you can assign a VLAN to a SSID and the AP will tag the traffic, so you do a trunk to the AP. For sanity, the trunk port default (untagged) VLAN should be the management VLAN, then tagged VLAN IDs for your client traffic.

 

[SycoPath]

OK i had a quick look - i think you havve a basic misunderstanding - yes ports can be dual mode (tagged and untagged traffic) but in your case for the ESXI servers they will be trunk ports - untagged traffic is simply traffic that comes from a device that does not understand VLANs and the switch port it is connected to (or the device such as the Unifi) is in charge of adding the tag - so in other words Tagged and Untagged has nothing to do with the packet on the wire it has to do with what the device at the end will do with it.

If you enable VLANs on the switch all packets that traverse the switch will have VLAN tags on them - some of them might be for the default VLAN for the switch - and when they get to an egress port - the VLAN tag will be stripped.

On a device such as a Unifi - assuming it supports both tagged and untagged clients - then the port the wireless clients connect to will have a PVID - any clients that do not put a tag on their packets (most PCs, phones etc) will have the PVID given to the packet

Craig

I have absolutely no experience setting up VLANS, so misunderstanding on my end is highly likely. I made a quick and dirty topology in paint for visual aid.

OK, so the switch tags the traffic coming in on a port from a device that doesn't understand vlans. How do I configure a port to pass traffic with VLAN10 and also no vlan (default vlan1)? Everything will still have the vlan bit set to 1 by default, correct?

Unifi allows me to configure an SSID to be on a network tagged with vlan10. So in my case, this traffic will already be arriving at the switch with the vlan10 tag on it, but there will also be traffic on the same switch port (1/1/46 and 1/1/48) with no vlan tags set. I need this traffic to flow, both vlan10 and no vlan, from 1/1/46 (Access Point 1) and 1/1/48 (Access Point 2) to 1/3/1 (trunk1 to ESXi vSwitch) and 1/3/6 (trunk2 to ESXi vSwitch) and to any other clients attached to the ICX6610, assuming they are on the same VLAN.

The whole point of this is network segregation, so I can do some parental controls on my kids network, and also to learn how VLAN's work. I've watched a few YouTube videos, but apparently it's left me with some incorrect assumptions.

According to this link, setting a vSwitch to VLAN 4095 will make it pass all vlan traffic without changing any tags, so if I can get tagged traffic leaving from the ICX6610 then pfSense can handle it from there, and I can have it handle DHCP separately from my default traffic as it should arrive at pfSense tagged VLAN10.

Does my configuration make sense?
What command makes a port into a trunk port on the ICX6610?
If traffic leaves a trunk port headed for pfSense, does it strip the VLAN or pass it still tagged to a vlan?
Do I need to configure any tagging on any ports except 1/1/25 and 1/1/26 for VLAN10? These ports will be desktops with no tag, but the switch should be adding VLAN10 to packets and routing to pfSense

 

[Craig Curtin]

I have absolutely no experience setting up VLANS, so misunderstanding on my end is highly likely. I made a quick and dirty topology in paint for visual aid.

OK, so the switch tags the traffic coming in on a port from a device that doesn't understand vlans. How do I configure a port to pass traffic with VLAN10 and also no vlan (default vlan1)? Everything will still have the vlan bit set to 1 by default, correct?

Unifi allows me to configure an SSID to be on a network tagged with vlan10. So in my case, this traffic will already be arriving at the switch with the vlan10 tag on it, but there will also be traffic on the same switch port (1/1/46 and 1/1/48) with no vlan tags set. I need this traffic to flow, both vlan10 and no vlan, from 1/1/46 (Access Point 1) and 1/1/48 (Access Point 2) to 1/3/1 (trunk1 to ESXi vSwitch) and 1/3/6 (trunk2 to ESXi vSwitch) and to any other clients attached to the ICX6610, assuming they are on the same VLAN.

The whole point of this is network segregation, so I can do some parental controls on my kids network, and also to learn how VLAN's work. I've watched a few YouTube videos, but apparently it's left me with some incorrect assumptions.

According to this link, setting a vSwitch to VLAN 4095 will make it pass all vlan traffic without changing any tags, so if I can get tagged traffic leaving from the ICX6610 then pfSense can handle it from there, and I can have it handle DHCP separately from my default traffic as it should arrive at pfSense tagged VLAN10.

Does my configuration make sense?
What command makes a port into a trunk port on the ICX6610?
If traffic leaves a trunk port headed for pfSense, does it strip the VLAN or pass it still tagged to a vlan?
Do I need to configure any tagging on any ports except 1/1/25 and 1/1/26 for VLAN10? These ports will be desktops with no tag, but the switch should be adding VLAN10 to packets and routing to pfSense

OK so lets take it step by step

A port is a trunk port on a 6610 by adding more than 1 VLAN to it (or in ICX parlance) adding it to more than 1 VLAN

Usually the 4095 VLAN is used for management traffic (by convention - not by hardcoding)

If you have a Unifi connected to a switch port - where is the untagged traffic coming from ?? (ports 1/1/46 and 1/1/48 ?) Turn them into trunk ports by setting all the VLANs they will be connected to. But yes you can make them dual mode and have them also handle untagged traffic - buts makes no sense.

In the case of something like PFsense you want to configure VLANs on PFsense and then attach a single port to your 6610 - this port would be marked as a trunk with all of the VLANs that you would want PFsense to see (if Pfsense is virtual (would not recommend that for a beginner)) then you would assing a single tagged interface through a vswitch connected to the ESXi trunk port

So usually what you would do on your Unifi is setup multiple SSIDs - lets call them Guest, Kids, IOT, Parents - you would assign each of these to a VLAN - lets call them


Guest = VLAN100
Kids=VLAN200
IOT=VLAN300
Parents=VLAN1000

On the switch port that the Unifi connects to you would do the following

conf t
vlan 100 name Guest
exit
vlan 200 name Kids
exit
vlan 300 name IOT
exit
vlan 1000 name Parents
exit

vlan 100
tag e 1/3/1
tag e 1/3/6
tag e 1/1/46

vlan 200
tag e 1/3/1
tag e 1/3/6
tag e 1/1/46
etc etc

This would make the ports 1/3/1 and 1/3/6 and 1/1/46 into trunk ports and able to see and process all traffic for those VLANs.

All of that traffic would be passed on a trunk port to the Pfsense box to be routed/firewalled based on your criteria.

Craig

 

[patg84]

Can a larger than 8gb CF card be installed in the VDX6740?

 

[Craig Curtin]

OK - considering the rear ports are not intended to be used for data ports, you won't find any information on this in the manuals etc. Using them as just data ports is easy enough, but splitting them up and using some for stacking and some for data ports requires some finangling to make the stacking code not try and take over all 4 ports.

However it's possible and stable across reboots etc. basically follow this, it might take you a couple times. This will enable stacking between two units just using the 40gbE qsfp ports, leaving 4x breakout QSFPs total available for data use

#unplug all stacking cables except for 1, going between port 1/2/1 on unit 1 and 1/2/1 on unit two
#don't plug anything else into unit 2 or you'll get a loop
#first you have to unstack all the units and remove any existing stack configuration:
enable
stack unconfigure all
write mem
reload
#both switches should reload, wait till they come back up

#when they're back up, on unit 1:
enable
conf t
stack unit 1
no stack-trunk 1/2/1 to 1/2/2
stack-port 1/2/1 1/2/6
#ignore the warning that pops up about not showing up in "show run"

#still running the below on unit 1!
stack unit 2
no stack-trunk 2/2/1 to 2/2/2
stack-port 2/2/1 2/2/6
#ignore the warning that pops up about not showing up in "show run"
exit
stack enable
write mem
exit
stack secure-setup

#When the setup is done, do "show run"
#stack unit 2 section probably has "stack-trunk 2/2/1 to 2/2/2" added back to it

#the second switch is currently rebooting to rejoin the stack, wait for it to do so
#once it comes back online, do the following to remove the stack unit 2 trunk:
#still on unit 1:
conf t
stack unit 2
no stack-trunk 2/2/1 to 2/2/2
#will probably reboot unit 2 again
#wait for it to come back online, then connect the second pair of 40gbE ports

Once you connect the second pair of 40gbe ports (1/2/6 on unit 1 to 1/2/6 on unit 2) you should be done. Check by running "show stack", you should see both units "ready" with the connection diagram showing both 40gbE ports linked up like below:

ICX1#sh stack
T=6m28.6: alone: standalone, D: dynamic cfg, S: static
ID   Type          Role    Mac Address    Pri State   Comment
1  S ICX6610-48P   active  cc4e.24b8.d9d0 128 local   Ready
2  S ICX6610-48P   standby cc4e.243e.aa74   0 remote  Ready

    active       standby
     +---+        +---+
-2/6| 1 |2/1--2/1| 2 |2/6-
|   +---+        +---+   |
|                        |
|------------------------|
Standby u2 - protocols ready, can failover
Current stack management MAC is cc4e.24b8.d9d0

If you have the same output, do a "write mem" then a "reload" to reload the whole stack. It should come back up fresh in the correct state and running "show stack" again should have the same output as before.

Your final stacking config should look like this:

stack unit 1
  module 1 icx6610-48p-poe-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
  priority 128
  stack-port 1/2/1 1/2/6
stack unit 2
  module 1 icx6610-48p-poe-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
  stack-port 2/2/1 2/2/6
stack enable

@fohdeesha - doing this will allow me to use the other two 40GB slots for straight connection (not fanout) to my Connect X3 cards ? So i could use 2 ports on the first switch for ESXi hosts and one on the 2nd switch for another host ? Or does this config lock the ports into fanout mode etc ?

Craig

 

[fohdeesha]

@fohdeesha - doing this will allow me to use the other two 40GB slots for straight connection (not fanout) to my Connect X3 cards ? So i could use 2 ports on the first switch for ESXi hosts and one on the 2nd switch for another host ? Or does this config lock the ports into fanout mode etc ?

Craig

the breakout ports will always be breakout ports, and the stack connection between switches must always contain 1 full 4gb port. so you're left with 1 40gb port available per switch