Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[klui]

Is there any hardware difference between front-to-back and rear-to-front airflow ICX6610s?

i.e. can I just open the fan modules and PSUs and turn the fans around?

I've got a Ports-to-PSU one here and I'd like to make it a PSU-to-Ports one. Does anyone have a PSU-to-Ports one and would mind sharing some pictures of the fan tray?

After reading @kapone's reply this piqued my interest. If I read your request correctly, you want to convert an (E)xhaust fan to an (I)ntake one.

If that is the case, then you should be able to do it without the surgery @kapone mentioned. Take a look at the pictures I just snapped of my exhaust fans and you should understand why.

The tricky part would be reinstalling the plastic pins on the bottom but possible using some long tweezers. Taking them out is easy. Just be aware the locking tabs would have splayed out after being in the lock position for so long. Reinstall them from the rear towards the front and you should have enough clearance. You should also have enough slack for PSU fans because the fans are in the same orientation.

I've read on this thread Intake switches are louder than Exhaust, but unsure if that's related to the PSU or combination of PSU/fan.

 

[mantis454]

Anyone else have higher that expected power use?

I have just acquired a ICX6450-24p and the idle power usage with nothing connected is about 60W (at 240V). With 1 x 10G and 6 x 1G (and supplying about 25W of POE) the power usage increases to about 85 - 89 W which is expected. However this still seems higher than others. The PSU does make a slight high pitch whine, maybe it has started to fail?

 

[rootpeer]

Hi everyone, I have some questions about stacking.

I have an ICX6450 as my main switch. I recently bought an ICX6610 to put in my "server closet" as a secondary but 40Gb capable switch.

I want to use the ICX6450 as the master/uplink switch and have all its settings propagate to the ICX6610.

Is that something that can be done? Or does the 6610 have to be the "core" switch? Or am I misunderstanding something?

Can you give me an idea of what I have to do to configure the stack?

 

[infoMatt]

I have just acquired a ICX6450-24p and the idle power usage with nothing connected is about 60W (at 240V).

That's strange, unless your meter is measuring the apparent power... at low load I've seen that the PSU has a slightly poor power factor; mine draws around 30-35W of real power (ie. the one that the power company bills the user for, at least for domestic usage here in Italy) at idle w/o anything connected.
I was expecting a lower consumption (the 25ish W written on the technical datasheet) but oh well.

The PSU does make a slight high pitch whine, maybe it has started to fail?

It might or not, a coil whine is not a real failure sign by itself (could very possibly be the silicone glue that has dried up and it's less dampening). On mine however the high pitch sound came from the fans, not the PSU...

 

[mantis454]

That's strange, unless your meter is measuring the apparent power... at low load I've seen that the PSU has a slightly poor power factor; mine draws around 30-35W of real power (ie. the one that the power company bills the user for, at least for domestic usage here in Italy) at idle w/o anything connected.
I was expecting a lower consumption (the 25ish W written on the technical datasheet) but oh well.

I think you're right. Just tried it with a TP Link Kasa smart plug with energy monitoring and it reads just about 20W lower all the time. Noticed the original power meter I used had a power factor readout that seems to be 1 all the time so it may not be that accurate. Thanks

 

[J--]

My Kill-A-Watt is reading 39w for a ICX6450-24P, just wiped.

 

[Rttg]

I have just acquired a ICX6450-24p and the idle power usage with nothing connected is about 60W (at 240V).

Yeah, I’m seeing about that wattage displayed on a Kill-a-Watt for my 6450-48p running on 120V - sounds like a wonky power meter

 

[ljvb]

Stupid question.. is it possible to bridge two ports on the 6610... I'm trying to simplify my network a little.. my primary gateway uses a dual gig card passed through via pci-passthrough. I am trying to migrate off that vmware server, so I want to connect my incoming gig connection (FIOS) to the switch, and then pass that through to a host which has a new gateway but uses virtual interfaces, on a private vlan.

 

[LodeRunner]

Stupid question.. is it possible to bridge two ports on the 6610... I'm trying to simplify my network a little.. my primary gateway uses a dual gig card passed through via pci-passthrough. I am trying to migrate off that vmware server, so I want to connect my incoming gig connection (FIOS) to the switch, and then pass that through to a host which has a new gateway but uses virtual interfaces, on a private vlan.

I take the Ethernet handoff from my ISP straight to my switch in a L2 VLAN that's trunked to my hosts. I haven't used PVLAN, so can't help you there.

 

[infoMatt]

Stupid question.. is it possible to bridge two ports on the 6610... I'm trying to simplify my network a little.. my primary gateway uses a dual gig card passed through via pci-passthrough. I am trying to migrate off that vmware server, so I want to connect my incoming gig connection (FIOS) to the switch, and then pass that through to a host which has a new gateway but uses virtual interfaces, on a private vlan.

Yep, use a spare VLAN without a router interface defined on it, use it as the native/untagged VLAN (if your ethernet handoff doesn't require a tagged VLAN, I don't know the details of the FiOS offer) on a port connected to the ISP and as a tagged VLAN on the router-on-a-stick

 

[gsrfan01]

Tossed this in its own thread but realized I should probably of just just it here..

Snagged an ICX6610 last year due to the popular thread here and my passing familiarity with them at a previous job. I'm using all but 1 of my 10g SFP+ ports up front, so when I bought another pair of ESXi nodes I wanted to use one of the QSFP+ breakout ports on the rear, but I'm running into some oddities. I'm using 1/2/6 as a 40g port to TrueNAS with no issues and had used 1/2/1 previously without trouble.

I've got 2 Cisco QSFP+ to SFP+ breakout cables, I believe they're passive but I'm not 100% sure on the model number, but can probably dig it up if that's helpful.

When I plug into the top QSFP+ it'll link up port 1/2/5 but not 1/2/2 - 1/2/4, only port 5. When I plug into the bottom QSFP+ port none of the ports 1/2/7 to 1/2/10 light up. They did briefly earlier today for a little bit but stopped seemingly after I added them to a VLAN, but I couldn't get them back up even after reverting the changes. I swapped between the 2 breakout cables and the behavior was the same. Also tried plugging the SFP+ ports 1-3 into known good hosts and they didn't link, but they work fine when connected to the front SFP+ ports.

I'm curious if it's likely an issue with the cable or I'm missing something else. I did order a 10gTek QSFP+ to SFP+ breakout cable that'll be in tomorrow.

Here's snippets of my running config for the pertinent ports:
Current configuration:
!
ver 08.0.30tT7f3
!
stack unit 1
module 1 icx6610-48p-poe-port-management-module
module 2 icx6610-qsfp-10-port-160g-module
module 3 icx6610-8-port-10g-dual-mode-module
!
global-stp
...
interface ethernet 1/1/48
port-name Uplink to Firewall
dual-mode 1001
!
interface ethernet 1/2/6
dual-mode
!
interface ethernet 1/3/1
port-name Ravager ESXi VM Uplink
dual-mode
speed-duplex 10G-full

It looks like it's being detected in show media:

Port 1/2/1: Type : EMPTY
Port 1/2/2: Type : EMPTY
Port 1/2/3: Type : EMPTY
Port 1/2/4: Type : EMPTY
Port 1/2/5: Type : EMPTY
Port 1/2/6: Type : 40GE-Passive Copper
Port 1/2/7: Type : 40GBASE-Passive Copper
Port 1/2/8: Type : 40GBASE-Passive Copper
Port 1/2/9: Type : 40GBASE-Passive Copper
Port 1/2/10: Type : 40GBASE-Passive Copper

 

[iotapi322]

Yep, use a spare VLAN without a router interface defined on it, use it as the native/untagged VLAN (if your ethernet handoff doesn't require a tagged VLAN, I don't know the details of the FiOS offer) on a port connected to the ISP and as a tagged VLAN on the router-on-a-stick

@infoMatt - I'm trying to do something like this....
I have the SB8200 cable modem that supports a static link aggregation and I want to send those two ports to cable modem, then have the "output" of that going to a VLAN on one of my 10GB ports.

Any tutorial on the internet for the brocade to help set that up?

 

[gsrfan01]

@infoMatt - I'm trying to do something like this....
I have the SB8200 cable modem that supports a static link aggregation and I want to send those two ports to cable modem, then have the "output" of that going to a VLAN on one of my 10GB ports.

Any tutorial on the internet for the brocade to help set that up?

Does it support LACP?

If so, probably something like this:

#Create lag
LAG "WAN" dynamic id 1
ports eth 1/1/47 to 48
primary-port 1/1/47
deploy
#Create VLAN 1003 for WAN
vlan 1003 name "WAN"
#Set modem ports to untag
untag eth 1/1/47 to 48
#Tag ports to 10g
tag eth 1/3/8

 

[dennisp]

Does it support LACP?

If so, probably something like this:

#Create lag
LAG "WAN" dynamic id 1
ports eth 1/1/47 to 48
primary-port 1/1/47
deploy
#Create VLAN 1003 for WAN
vlan 1003 name "WAN"
#Set modem ports to untag
untag eth 1/1/47 to 48
#Tag ports to 10g
tag eth 1/3/8

That looks like it will work, but try changing the lag to static if it won't come up with dynamic. Also check to see if you want the 10g port to be tagged or untagged to the new gateway

 

[iotapi322]

No it doesn't support dynamic has to be static....

This is the part i don't understand...

#Tag ports to 10g

tag eth 1/3/8

How do you 'route' all the data from 47/48 to 1/2/4 ( 10G port). Do i just tag ports 1/1/47 - 1/1/48 and port 1/2/4 with the VLAN ?

 

[gsrfan01]

No it doesn't support dynamic has to be static....

This is the part i don't understand...

#Tag ports to 10g

tag eth 1/3/8

How do you 'route' all the data from 47/48 to 1/2/4 ( 10G port). Do i just tag ports 1/1/47 - 1/1/48 and port 1/2/4 with the VLAN ?

You'd have to pipe that traffic somewhere, I assumed the 10g port uplinks to a PFsense box / VM or something of that nature which would actually route.

Depending on what you're using for a firewall and how it's setup will change if it's tagged or untagged.

 

[iotapi322]

You'd have to pipe that traffic somewhere, I assumed the 10g port uplinks to a PFsense box / VM or something of that nature which would actually route.

Depending on what you're using for a firewall and how it's setup will change if it's tagged or untagged.

Yes I have the pfsense setup to have the WAN interface to be ix0.10 and the LAN to be ix0

I'm just not clear on how to segregate the cable modem traffic on ports 47 / 48 from being on the rest of the switch.

 

[gsrfan01]

Yes I have the pfsense setup to have the WAN interface to be ix0.10 and the LAN to be ix0

I'm just not clear on how to segregate the cable modem traffic on ports 47 / 48 from being on the rest of the switch.

In that case it sounds like ix0.10 is VLAN10 on the WAN port, so you'd want to do a
vlan 10
tag eth 1/1/47 to 1/1/48
to add those to VLAN 10.

A VLAN is a virtual LAN, like carving out ports on the switch. Anything in VLAN10 would be on the same layer 2 network as your modem, but only devices in that VLAN would see that traffic.


Do you have other ports on VLAN10 in the switch? Additionally, what's the topology for the LAN interface?

 

[iotapi322]

In that case it sounds like ix0.10 is VLAN10 on the WAN port, so you'd want to do a
vlan 10
tag eth 1/1/47 to 1/1/48
to add those to VLAN 10.

A VLAN is a virtual LAN, like carving out ports on the switch. Anything in VLAN10 would be on the same layer 2 network as your modem, but only devices in that VLAN would see that traffic.


Do you have other ports on VLAN10 in the switch? Additionally, what's the topology for the LAN interface?

Everything is flat right now, I'm setting up the VLAN because I want to only have one interface card in my router, but I have to have the link aggregation in the cable modem to get the full 1.2 GB/s

So I've run the following

vlan 10 name "CM_WAN"
untag eth 1/1/47 to 1/1/48
show vlan


Total PORT-VLAN entries: 2
Maximum PORT-VLAN entries: 64

Legend: [Stk=Stack-Id, S=Slot]

PORT-VLAN 1, Name DEFAULT-VLAN, Priority level0, Spanning tree Off
 Untagged Ports: (U1/M1)   1   2   3   4   5   6   7   8   9  10  11  12
 Untagged Ports: (U1/M1)  13  14  15  16  17  18  19  20  21  22  23  24
 Untagged Ports: (U1/M1)  25  26  27  28  29  30  31  32  33  34  35  36
 Untagged Ports: (U1/M1)  37  38  39  40  41  42  43  44  45  46
 Untagged Ports: (U1/M2)   1   2   3   4
   Tagged Ports: None
   Uplink Ports: None
 DualMode Ports: None
 Mac-Vlan Ports: None
     Monitoring: Disabled

PORT-VLAN 10, Name CM_WAN, Priority level0, Spanning tree Off
 Untagged Ports: (U1/M1)  47  48
   Tagged Ports: None
   Uplink Ports: None
 DualMode Ports: None
 Mac-Vlan Ports: None
     Monitoring: Disabled

I think i need to set the cable modem link aggregation ports to dual mode.... Not sure.

 

[gsrfan01]

I think i need to set the cable modem link aggregation ports to dual mode.... Not sure.

That'd depend on the modem, here's the SB8200 config, looks like it's LACP.

You'd want to tag your 10g port as well, something like
vlan 10
tag eth 1/3/1

That would get your WAN link online for PFsense. It looks like your LAN is untagged, are you intending to have the WAN/LAN share the same 10g port?