Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[fohdeesha]

look mom I made an opnsense juniper

 

[Craig Curtin]

the breakout ports will always be breakout ports, and the stack connection between switches must always contain 1 full 4gb port. so you're left with 1 40gb port available per switch

Aah - OK thats a bummer then - looks like i might leave it as they are and connect up a new Juniper EX4300 i have coming

Craig

 

[guyman]

Hello, was scrolling through this thread a bit and decided to pull the trigger on a 6450-24p (and ended with an extra free 6610-24p). Highly appreciative of all the great information here along with the licencing option. Followed the guide and got my 6450 working beautifully, only concern is power draw. With little to no traffic, one sfp dac to my pc and one sfp 10g tranciever to my router, according to my Kill A Watt I am drawing 50w (around double the estimate) Is this normal?

On another note I ordered Delta EFB0412VHD F00 fans to install, and I am also considering dremeling off the fan grate as mentioned in another post above to quiet the unit even more.

 

[NablaSquaredG]

With little to no traffic, one sfp dac to my pc and one sfp 10g tranciever to my router, according to my Kill A Watt I am drawing 50w (around double the estimate) Is this normal?

25W in the start post doesn't seem to be correct, as I've verified with at least 5 different 6450-24P. They should IDLE around 35W... I only tested with 240V though

 

[guyman]

25W in the start post doesn't seem to be correct, as I've verified with at least 5 different 6450-24P. They should IDLE around 35W... I only tested with 240V though

Hmm, that sucks, oh well. Maybe the new fans will shed a few watts. Wonder if the power supply isnt the most efficient either...

 

[creese]

Hi all,

I recently acquired a pair of ICX 6450-24P's and have a few questions I hope some of you might be able to help with:

1. SSH Connectivity Issue: I've set these up as per the guide, and everything works fine except for SSH access. Whenever I try to SSH into a switch, the connection times out. Surprisingly, I couldn't find any relevant logs about the SSH attempts on the switch. What troubleshooting steps would you recommend?
2. Stacking Configuration: The guide I followed touched on most topics but left stacking as a pending item. Could someone provide insights or resources on how to do this?
3. Network Topology: On a slightly different note, I have a server with dual NICs I'm considering bonding. Is it better to connect these to different switches for redundancy? If I choose this setup, would I still be able to bond the NICs?

Thanks!

 

[kpfleming]

Network Topology: On a slightly different note, I have a server with dual NICs I'm considering bonding. Is it better to connect these to different switches for redundancy? If I choose this setup, would I still be able to bond the NICs?

Yes, if the switches are stacked, they act as a single unit and any ports on them can be included in LAGs. If you want redundancy, then splitting the server's ports across the two physical switches will provide it.

 

[Craig Curtin]

Hi all,

I recently acquired a pair of ICX 6450-24P's and have a few questions I hope some of you might be able to help with:

1. SSH Connectivity Issue: I've set these up as per the guide, and everything works fine except for SSH access. Whenever I try to SSH into a switch, the connection times out. Surprisingly, I couldn't find any relevant logs about the SSH attempts on the switch. What troubleshooting steps would you recommend?
2. Stacking Configuration: The guide I followed touched on most topics but left stacking as a pending item. Could someone provide insights or resources on how to do this?
3. Network Topology: On a slightly different note, I have a server with dual NICs I'm considering bonding. Is it better to connect these to different switches for redundancy? If I choose this setup, would I still be able to bond the NICs?

Thanks!

Are you sure you followed the guide by the OP in this thread - i have done about 6 of the 6450 and not had a single issue with SSH (other than Putty whinging that the cipher is old)

In the same Guide fohdeesha shows how to turn off stacking.

Failing that check the Youtube channel for the Brocade switch config as there is one guide in there on it.

Craig

 

[solloron1]

Hi all,

I have a 7150-c12p and I'm trying to get my guest VLAN ACL sorted. I can make it so the VLAN cannot get out to any other RFC1918 subnet but how can I make it so the main LANs can access them?

Config snippets:

vlan 40 name Guest by port 
 tagged ethe 1/1/11 to 1/1/12 ethe 1/2/1 
 router-interface ve 40 
 spanning-tree 802-1w 
 ip access-group 140 in

ip access-list extended 140
 sequence 10 deny ip 192.168.40.0 0.0.0.255 10.0.0.0 0.255.255.255
 sequence 20 deny ip 192.168.40.0 0.0.0.255 172.16.0.0 0.15.255.255
 sequence 30 deny ip 192.168.40.0 0.0.0.255 192.168.0.0 0.0.255.255
 sequence 40 permit ip 192.168.40.0 0.0.0.255 any

I seem to be missing something. Even if I add permit rule before the deny it allows the guest to access the LAN as well. Thanks

 

[SycoPath]

Huge thanks for the replies. This has been a major help in my understanding of how this all works.

OK so lets take it step by step

A port is a trunk port on a 6610 by adding more than 1 VLAN to it (or in ICX parlance) adding it to more than 1 VLAN

Usually the 4095 VLAN is used for management traffic (by convention - not by hardcoding)

I mentioned vlan 4095 because VMWare states if you set a vSwitch to vlan 4095 it will "pass all traffic unmolested including vlan tags" according to the article I linked, so I assumed this is how to configure a trunk port for a vSwitch since my pfsense is actually virtualized under ESXi. I thought the ESXi vSwitch would drop packets incoming tagged to a vlan it was not configured to be a member of. Is this correct?

If you have a Unifi connected to a switch port - where is the untagged traffic coming from ?? (ports 1/1/46 and 1/1/48 ?) Turn them into trunk ports by setting all the VLANs they will be connected to. But yes you can make them dual mode and have them also handle untagged traffic - buts makes no sense.

UNTAGGED traffic is anything without a vlan tag, correct? Everything from my primary SSID has no vlan tags right now, and is leaving the Unifi AP without vlan tags, making it UNTAGGED traffic, correct? My switch is configured to have the default vlan be VLAN1, so all traffic coming into the switch without vlan tags is considered by the switch, to be VLAN1, correct? Currently all switch ports are members of VLAN1. This also means that when traffic leaves the switch it will be UNTAGGED, correct?

In the case of something like PFsense you want to configure VLANs on PFsense and then attach a single port to your 6610 - this port would be marked as a trunk with all of the VLANs that you would want PFsense to see (if Pfsense is virtual (would not recommend that for a beginner)) then you would assing a single tagged interface through a vswitch connected to the ESXi trunk port

So usually what you would do on your Unifi is setup multiple SSIDs - lets call them Guest, Kids, IOT, Parents - you would assign each of these to a VLAN - lets call them


Guest = VLAN100
Kids=VLAN200
IOT=VLAN300
Parents=VLAN1000

I plan on adding more VLANs, each with it's own DHCP and /24 IP space restricting inter-vlan communication by utilizing pfsense firewall rules later. Once I wrap my head around setting up a single vlan and getting it working, I'll be adding more. I figured getting a single VLAN working correctly would be a lot easier than trying to get 5-6 working all at once.

On the switch port that the Unifi connects to you would do the following

conf t
vlan 100 name Guest
exit
vlan 200 name Kids
exit
vlan 300 name IOT
exit
vlan 1000 name Parents
exit

vlan 100
tag e 1/3/1
tag e 1/3/6
tag e 1/1/46

vlan 200
tag e 1/3/1
tag e 1/3/6
tag e 1/1/46
etc etc

This would make the ports 1/3/1 and 1/3/6 and 1/1/46 into trunk ports and able to see and process all traffic for those VLANs.

All of that traffic would be passed on a trunk port to the Pfsense box to be routed/firewalled based on your criteria.

Craig

This helps a lot and is definitely what I was asking someone to share with me. Adding a port to be a member of multiple vlans will let traffic from ANY of the configured member vlans egress out of that port. In the configuration above, all packets leaving 1/3/1 will retain it's VLAN tag, correct? So from the above example, packets would egress 1/3/1 with VLAN100, VLAN200, VLAN300, and VLAN1000 tags intact headed for ESXi's vSwitch, correct? Also, setting the dual-mode for ports would allow all member VLAN traffic to flow, as well as UNTAGGED traffic, correct?

I currently have pfSense connected to the ESXi port group 4095, as this is how I thought trunk ports are set up for vSwitches, and this seems to be working as expected, since I still have an internet connection (See attached image).

 

[tr_deal]

First I want to give a huge shout out to fohdeesha! The guides and this thread we're the reasons I picked up a 6610 and integrated it into my homelab.

I have been banging my head on one thing and was wondering if anyone had any suggestions. I have 4 Amcrest PoE cameras on vLAN50. No matter what I do the switch only negotiates 100full to the cameras. If I try to force 1000master I lose connection.

I've upgraded the firmware on the cameras to the latest and verified when I move them to my Aruba 2930m that switch negotiates them to 1000full. The only thing that keeps sticking out to me is when I look at the Inline Power Statistics for the camera ports it shows the correct type (802.3af) but for class it says n/a. On the ports where my WAPs are connected is has both the correct type and class (Class 3). This may be nothing but its the only difference I see.

Any thoughts or pointers would be appreciated!

 

[LodeRunner]

First I want to give a huge shout out to fohdeesha! The guides and this thread we're the reasons I picked up a 6610 and integrated it into my homelab.

I have been banging my head on one thing and was wondering if anyone had any suggestions. I have 4 Amcrest PoE cameras on vLAN50. No matter what I do the switch only negotiates 100full to the cameras. If I try to force 1000master I lose connection.

I've upgraded the firmware on the cameras to the latest and verified when I move them to my Aruba 2930m that switch negotiates them to 1000full. The only thing that keeps sticking out to me is when I look at the Inline Power Statistics for the camera ports it shows the correct type (802.3af) but for class it says n/a. On the ports where my WAPs are connected is has both the correct type and class (Class 3). This may be nothing but its the only difference I see.

Any thoughts or pointers would be appreciated!

Try just '1000-full' instead of '1000-full-master'? Or is just '1000-full' not available on the 6610?

 

[bwahaha]

it shows the correct type (802.3af) but for class it says n/a

I'm pretty stoned right now, but I can't seem to find any way to set the class on a port via cli. Ironically, it's right there in the web gui. Might try setting those ports to an appropriate class there.

chart for easyish reference:

Power over Ethernet - Wikipedia

en.wikipedia.org

 

[klui]

Try enabling LLDP per https://forums.servethehome.com/ind...be-40gbe-switching.21107/page-147#post-250823

Otherwise, override POE class through inline power ethernet <INTERFACE> power-by-class <0-4>

 

[tr_deal]

Thanks for the suggestions! Unfortunately the ports are still only negotiating to 100-full.

 

[VirtualBacon]

Eyeing up a ICX7150-C12P. Do they block unsupported transceivers? I'll be ordering from FS.COM anyway, but I can't seem to find an answer

 

[LodeRunner]

Eyeing up a ICX7150-C12P. Do they block unsupported transceivers? I'll be ordering from FS.COM anyway, but I can't seem to find an answer

As far as I recall, it won't stop you from using 3rd party optics of any sort, but the optical monitoring will be disabled by default for non Brocade optics and can be overridden with a console command. Of course, if you're ordering from FS.com just have them code the optics or DACs as Brocade. There's no secret key being used.

 

[patg84]

As far as I recall, it won't stop you from using 3rd party optics of any sort, but the optical monitoring will be disabled by default for non Brocade optics and can be overridden with a console command. Of course, if you're ordering from FS.com just have them code the optics or DACs as Brocade. There's no secret key being used.

Does this apply to the VDX line?

 

[Midvalley]

Hello all. Just wanted to drop a message to thank everyone for the information in this thread to date. I stumbled across this a couple weeks back while I was investigating PoE switches to power some IP cameras I had just purchased. Without finding this, I don't know if I would have started down the road of used enterprise gear and probably ended up with a prosumer switch of some sort. However, based on the info here, I was able to search out a used ICX6450-48P for a pretty good price. The switch arrived today and is in pretty good condition all in all, and I'm currently working through the initial config and updates.

Kudos to @fohdeesha for the documentation and resources, it's all really well written and concise. Also, kudos to the community that has sprung up in this thread, there is a ton of information I am sifting through while I go about my first foray into enterprise networking.

I'll probably be around asking borderline dumb question that may have already been answered soon, so I figured I should have something other than a "how do I" question as my first post on STH.

Thanks all!

 

[tr_deal]

Hey everyone, I have another odd situation and after trying to search all 442 pages of the thread didn't find anything helpful...

I have 2 PSU and 2 Fan Trays in my 6610. One of the fans in my fan trays has a bad bearing, so I bought a new fan tray off eBay. However, when I swap the new fan tray in the fans never kick down from full speed. If I plug the old fan tray in its fans do kick down after a minute of so...

I followed the guide and updated all the firmwares, and cold booted the switch to ensure the IC2 bus reset. dm fan-speed shows fan tray 1 fans running at 21,000, while fan tray 2 fans are at 6000.

Any ideas?