Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

AlphaG

Active Member
Jun 8, 2017
102
26
28
53
I have a general network configuration question and have three 6450 switches in different locations because some parts of the house are old and others newer.

Currently internet comes into the house via cable company into a first floor closet. This is nowhere near the rest of the equipment so the open internet WAN connection travels up two stories to the attic above second floor, across 30 to 40 feet, then down two stories to a packed equipment cabinet that has mostly AV and automation gear. My pfsense box is in here. The LAN starts here with feed to the cabinet it’s in, and branching to other consumer switches via cat5e wires going to a second story new construction closet where my ‘home lab’ is, as well as back to the original closet where the cable came in to another small consumer switch that supplies the office computer next door to it.

The new setup has three 6450’s stacked, one in each of these locations, as soon as I run the fiber.

Can I segment off a vlan, say vlan999, for the WAN connection and eliminate my long cat5e run from the equipment cabinet to the router? Do I need to assign a vlan, say vlan10 to the remaining ports on the stacked switches if I do so? Or am I just asking for trouble?
 
  • Like
Reactions: tommybackeast

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
yeah pretty easily, I do the same at home. I would make normal traffic vlan 10 or something like you said, untagged on all ports, tagged on the ports between switches. Same with WAN, untagged on the port the ISP connects to, tagged on the ports between switches. That way the inter-switch links are only running tagged traffic and you don't have to deal with dual-mode. Just be sure to *not* put a VE or any other type of interface in the WAN vlan, or the ISP will see it as customer equipment instead of a passive link to your actual CPE

here's a snip of what mine looks like:


Code:
vlan 5 name WAN by port
 tagged ethe 1/3/8 ethe 2/3/8
 untagged ethe 1/1/1


vlan 10 name LAN by port
 tagged ethe 1/2/6 ethe 1/3/8 ethe 2/2/6 ethe 2/3/8
 untagged ethe 1/1/2 to 1/1/48 ethe 1/2/7 to 1/2/10 ethe 1/3/1 to 1/3/7 ethe 2/1/1 to 2/1/48 ethe 2/2/7 to 2/2/10 ethe 2/3/1 to 2/3/7
 router-interface ve 10
 

maes

Active Member
Nov 11, 2018
102
69
28
Just wanted to know how this project turned out.. Are the temps ok now? I tried to put the Fans in, but they stop working after a few minutes (no rotation at all)..
(...)
Ultimately my 6450-48p needs to POE (...)
I only have a 6450-24, non-POE. By nature it would generate a LOT less heat, but it also seems the fan voltage doesn't drop as much on the -24 model. The Noctuas never completely stop spinning in 'low speed' mode. However, they move so little air around that the temperatures start creeping up pretty quickly.

I will likely mod them to be permanently powered off the 12V line instead of letting the board control their speed. Even at full speed they're basically silent, so there's no point in running them slower.

I don't know what the 'default' fan configuration is on the -48p, but 3 noctuas are probably too little if it uses 2 fans normally. 2 of the noctuas together move just about as much air as the single 'factory' 40mm fan does on the -24.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
default fans on the 48P are 3, but with no PoE load it draws the same amount of power and puts out the same amount of heat as the 48 non-poe, increasing a few watts per PoE device depending. the three fans are sized to cool 770 watts of PoE load. if you have 4 or 5 2-watt PoE cameras you could run one stock fan, I have been for some time. These things don't mind running warm, the stock software doesn't even bother spinning the fans faster until 70 or 75c. 50c is nothing
 
Last edited:

Snorf

New Member
Nov 12, 2018
24
8
3
BC, Canada
Need a little help, sorry if this has been answered but I cannot find confirmation. I just want to make sure before ordering a bunch of stuff that may end up as useless.

Background: I have 2 – ICX6610-48P’s, and I have some HP 656089-001 Infiniband 10/40GB DP NIC 649281-B21 cards to be used in my server(s) and computer(s) at 10gbe and 40gbe.

I am not going to stack the 6610’s yet, but probably in the future.

Since I am not stacking yet I wanted to use the 40gbe ports on the 6610 for servers and comps. I have read the stacking useful hints and I understand I need Type B cables or Type A and flip the end as in the video with the paper clip to stack (all cables female x female). But……..(some FS links for clarity only)

Questions:

1. For 40gbe from ports 1/2/1 and 1/2/6 do you use Type A or Type B MTP/MPO cables plus transceivers to hook the 6610 to my NICs in a server/comp?

2. Would these cables work for stacking, and for 6610 to server/comp at 40gbe?

2.1. Mellanox MC2206130-001 40G QSFP+ DAC Cable

2.2. 1m (3ft) Mellanox MC2206310-001 Compatible 40G QSFP+ Active Optical Cable

3. Does it matter if you use 8 fiber or 12 fiber MTP/MPO cables to make these connections, 6610 to NIC, or for stacking?

4. If using an MTP/MPO to LC cassette does it matter about 8 fiber or 12 fiber MTP/MPO cables from 6610? I understand that the Cassette will say if it is flipped/Type A/B for hook-up. But in post #5 of this thread I am a little confused by the 24 port patch panel. It only has 4 MTP/MPO connections but breaks out to 24 SFP+ (12 fiber cable). I only have 16 ports of 10gbe available, do 8 ports just not work? Do I need to worry about 8 fiber vs 12 fiber MTP/MPO cables and how does it effect how/where I use them?

Thanks in advance for clearing this up for me.

Snorf
 

ljvb

Member
Nov 8, 2015
97
32
18
47
Quick question.. might beed to post this in the VMWare section, but I'll check here first. What is the hashing algorithm for VMware for the 6610 LAGs. Trunk type shows hashed based, but I have a ton of options when setting up the vDS, normally I picked source and destination based IP. Hopefully that is right.

One of my lags interfaces is showing inactive.
Port [Sys P] [Port P] [ Key ] [Act][Tio][Agg][Syn][Col][Dis][Def][Exp][Ope]
1/2/2 1 1 20001 Yes L Agg Syn Col Dis Def No Ina
1/2/3 1 1 20001 Yes L Agg Syn Col Dis No No Ope

The second lag shows both operational. Side note, this interface is also not showing up in VMWare (that is off topic here, but possibly related, I am going to post that in the VMWare section as I am having other problems with a second server).

Related to that server.. for some reason both 1/2/4 and 1/2/5 were showing down on both the switch and VMWare. I could not for the life of me figure out why, but pulling the power from the switch and restarting seems to have corrected that, but they keep going down after a few days. I am using the Avago 40/10 breakout cable linked much further up in the thread. Not sure what is happening. I had no issues when I used just two of the ports, now that I am using all 4, things have been going haywire. Now there is apparently a known issue with the intel X520-DA2 and the vsphere 6.7 update.. but the links remain down even when power cycling the server, they should at least show a link....

Dunno.. might have to dump lacp and just use 1 10G link in each box and a bunch of 1G links.
 

Joao Paulo Mello

New Member
Jan 16, 2019
14
1
3
Hi guys, it seems that I'm bit late for the party. Comparing ICX64xx prices (at ebay) from the first post to this one that seems crazy, you people definitely inflated the market and for me is even harder cos I have to pay $150 more for the USPS shipping if I got lucky.
But better late than never. I'm literally hunting for a ICX64xx-P. Personally I prefer a quiet 6450-P but 8x10GB SFP+ on the front of a 6610 is tempting. But I know noise. I'm a proud owner of ASA-5512x and a Nexus 3048 and I had to mod both to keep my life going forward without going insane. We all know, the place where we live aka home isn't a place those devices were designed to work. So we complain, whinings here and there but it is what it is. What we can do is to avoid these gears but I don't know. I hate them but I love them, so I try to acommodate things the way I can keep it bearable, people stash everything away on the basement like we all do with hammers and pliers to keep the heat and noise away. I can't keep my gear away from me. Most of my life I spent coding for a living systems that supports network infrastructure, so I have to interact with the equipment all the time. Currently I'm not actively working everyday, just decided to slow down a bit and spend some pings with the wife. I have now mostly Cisco gears, which is the last company I worked for and I still work sometimes as a consultant. The thing is, I want to install surveillance gear around the house and I'll get rid of my Nexus 3048TP and get a POE switch. I have 2 ESXi hosts and a NAS using the 4 SFP+ ports from the Nexus for storage. I need Layer3 at switch level to run simulations and other stuff. I read somewhere the 6450 doesn't support VRF-Lite, is that true?
I read some posts where people says 6610 is very noisy during initialization but the fans slow down after that just like most network gear. The ASA 5512x does the same but nothing near bearable, so I modded the fans using Noctua reducer cables, isn't dead silent but it's ok, the ASA doesn't complain and just get 4C/6C hotter.
So I'm thinking which one, risk everything for the looks and 16x10Gbe (6610) or keep it safe by choosing the safe heaven (6450).
before I go, thanks everyone for this thread specially to @fohdeesha. I'm always on STH, great articles and everything but when I read this thread I had to sign up.
Sorry by the mistakes I've made with the language. I lived for some years in US but it was long time ago.
Be safe.
 
  • Like
Reactions: mathiastro

DanielWood

Member
Sep 14, 2018
44
17
8
I read some posts where people says 6610 is very noisy during initialization but the fans slow down after that just like most network gear. The ASA 5512x does the same but nothing near bearable, so I modded the fans using Noctua reducer cables, isn't dead silent but it's ok, the ASA doesn't complain and just get 4C/6C hotter.
So I'm thinking which one, risk everything for the looks and 16x10Gbe (6610) or keep it safe by choosing the safe heaven (6450).
The 6610 never gets quiet, and there is not yet a way to silence it (replacing fans wont work because it expects minimum RPMs and fails to boot). If noise is a concern, avoid the 6610.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
the 6610 never gets silent/very quiet, but with rev B power supplies it should spin WAY down after boot, and you shouldn't be able to hear it behind a closet door. I suspect the people with very loud 6610's in this thread have rev A PSUs
 

Joao Paulo Mello

New Member
Jan 16, 2019
14
1
3
the 6610 never gets silent/very quiet, but with rev B power supplies it should spin WAY down after boot, and you shouldn't be able to hear it behind a closet door. I suspect the people with very loud 6610's in this thread have rev A PSUs
Rev B Dual PSU's right? Is there any way to predict the PSU's revision just by serial numbers or something? I mean, things that we can ask for the seller without the need to open the equipment? I'm willing to take a shot on 6610 with Rev B PSU's.
 

frameshift18

Member
Jan 9, 2019
31
24
8
This thread has convinced me to pick one of these up! I was concerned they would be extremely loud but it's actually a lot quieter than I would have thought based on some feedback on here. I do have a single rev c and the exhaust air direction.
 

frameshift18

Member
Jan 9, 2019
31
24
8
Rev B Dual PSU's right? Is there any way to predict the PSU's revision just by serial numbers or something? I mean, things that we can ask for the seller without the need to open the equipment? I'm willing to take a shot on 6610 with Rev B PSU's.
The revision number is printed on the sticker on the psu, all they'd have to do is unscrew 2 screws and pop it out real quick. It's not like they have to boot it up and run commands on the CLI to check or anything.
 

Joao Paulo Mello

New Member
Jan 16, 2019
14
1
3
Thanks @frameshift18. That's good to know I'll ask them and see how it goes.
Guys, there's this listing on ebay. 6610-P dual psu and it seems with 10G license installed, but too expensive IMO, even with free shipping to my country.
Brocade ICX 6610-48P ICX6610-48P-E 48-Port Gigabit PoE+ Managed Network Switch | eBay

Seller has 9 units. I hope prices starts coming down a little very soon.

Edit: @fohdeesha .
Can you please keep a 10G license for the 6610 hidden somewhere? I'll need one and hopefully very soon.
 
Last edited:

ljvb

Member
Nov 8, 2015
97
32
18
47
I figured out my problem.. I think.. It appears to be the breakout cable.. something funky is going on.. at least I hope it is the cable. I picked up a Cisco 40x10 breakout copper one which should arrive tomorrow. Guess I'll figure out whether it was the cable or not.
 

juey

Member
Oct 1, 2018
56
16
8
Germany
I figured out my problem.. I think.. It appears to be the breakout cable.. something funky is going on.. at least I hope it is the cable. I picked up a Cisco 40x10 breakout copper one which should arrive tomorrow. Guess I'll figure out whether it was the cable or not.
Must be, i have no problems at all with my 6610 connected to x520 using vsphere 6.7. Im using the breakout from FS.
 

Joao Paulo Mello

New Member
Jan 16, 2019
14
1
3
I know this was answered before but I couldn't find it. I'm almost pulling the trigger on that overpriced 6610-48p. The switch has 2 PSU's, that's great there's a chance fans will slow down after initialization ( the nexus 3048TP has the same behaviour. A banshee with one PSU, a smaller banshee with 2 ). The listed switch has only one fan cage. 2 fan modules helps in some way this picture? Maybe slow down the PSU's fan a little more?

Edit: The 6610-24p is quieter?
 
Last edited:

ljvb

Member
Nov 8, 2015
97
32
18
47
You all need to quit complaining about the noise.. You don't know noise till you have a couple of these in your garage rack.... The fans on the C3000 blade chassis are Boeings on take off.. all the time....