Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[dlaffin1012]

You configure the stack ports within the OS, you can pick which of the 40g ports to use on the back.

The 6610-24p take a little less juice than the 48p but if you have 2x power supplies in them you are going to have the same noise. I supposed you could run just one PS in each of the two. 2x 24p vs. 1x 48p are going to take more watts though.

You can always get 2x 6610s and keep one as a cold spare/powered off to swap in if the primary goes down instead of having it on all of the time. You won't have as many 10g/40g ports as 2x 24p's though. These are enterprise grade switches which are designed to run 24/7 so the odds of one going out is probably fairly slim. They have redundant power supplies and fans which can be swapped out if they go bad.

If it isn't mission critical I wouldn't worry about dual homing, LACP or any of that. If you have a spare 6610-48p (or whatever) powered off it is only going to take you a little bit of time to swap cables and power it on. The 2x 24p's would give you more redundancy assuming you dual home everything, but then again you are doubling your port requirements than single homing.

So at this point stacking the switches has redundancy built in but the main goal is more ports? I did ask before how often network cards die or ports die. the reason for that is I have done LACP mostly for redundancy more than performance so if a NIC died or a port died.

I at least want to connect the NAS and Backup Server to 40G. Everything else will be 10G or 1G ports. I do not need more than 48ports currently, I use about 20-26 ports right now. Stacking would be cool but at this point is it worth the extra money for that. Power isn't that big of an issue but my main thing if the redundancy isn't worth it then I would prefer not to use the extra power. As for other devices such as the NAS, Backup server, Main Server, Router, etc. that all have 10G or 40G capabilities, is it worth having LACP on those devices. My main server runs a decent amount of VM's, half of them being game servers and the others being house systems. same with the backup server and the NAS. Is it worth it to have LACP on those sytstems for redundancy or is the device likely to dies before a port or NIC dies?

If it isn't common for ports to die, then I amy just do one 6610-48p as it has dual psu's anyway so power is redundant at least. I would have two 40G ports to use and the other two breakout into 10G ports. At that point is it still worth doing LACP on the other devices then, since I would only have one switch. At 40G the NAS and backup server would not need the LACP for speed but more for redundancy. and rest of the systems LACP 10G would be also be more for redundancy as any single 10G or single 40G connection would be enough for any of the tasks at hand. I would prefer not to have a cold spare if it is not used and these are not known to die easily. If the LACP/Network Redundancy isn't worth it at this point then the only reason I would stack would be for pure enjoyment. I could either do two 24's or a single 48. If it were worth it I would assume the two 24' are better as I could stretch the LACP across the two switches for redundancy. If I go with the single 48 would the redundancy be worth doing on that let alone if the redundancy is even worth it in the first place.

 

[kapone]

just do one 6610-48p as it has dual psu's anyway so power is redundant at least.

Unless those two PSUs are connected to independent power circuits, there is no redundancy.

 

[dlaffin1012]

Unless those two PSUs are connected to independent power circuits, there is no redundancy.

Well I mean in the sense of if a psu dies the other keeps it going. It is on a UPS as well.

 

[kapone]

Well I mean in the sense of if a psu dies the other keeps it going. It is on a UPS as well.

Maybe...

PSUs don't die that easily, especially not of Brocade quality. If something killed one PSU and both PSUs were on the same circuit... the chances of the second one dying as well are pretty high.

 

[yukaia]

Maybe...

PSUs don't die that easily, especially not of Brocade quality. If something killed one PSU and both PSUs were on the same circuit... the chances of the second one dying as well are pretty high.

True dat, I'm actually fixing a PSU for my friend's ICX6450. It was on a UPS that failed and ended up killing everything that was plugged into it.

 

[dlaffin1012]

Hmm. Well I guess the only other reason to get dual psu is the fact that finding an extra isn't easy. But I still don't know which one to get and how many. I was originally going to lacp the 40G stuff but now it looks like single 40G connections is fine and single 10G connections is fine. From what I am learning it is uncommon that one of those connections would go down.

 

[ViciousXUSMC]

Got my two switches today.
ICX 6610-48P and ICX 6450-48P
Looks like I got lucky I have Rev 3 Power Supplies on the 6610.

No ears thats a major bummer. I will have to find some or make some. I do not like switches on a shelf where they can slide around.

Interesting the 6450 is wider than the 6610 side to side.
I'll power them up here soon and get some power draw figures and noise figures.

Edit: the 6450 is actually much louder than I expected. It had the old L2 firmware.
It quieted down after update but is still far from silent. It has 3 fans and is about 60db of sound, was almost 70db before the firmware update.

Power supply 3 not present

Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 ok, speed (auto): [[1]]<->2
Fan 3 ok, speed (auto): [[1]]<->2

Fan controlled temperature: 41.5 deg-C

Fan speed switching temperature thresholds:
                Speed 1: NM<----->65       deg-C
                Speed 2:       56<-----> 79 deg-C (shutdown)

Sensor B Temperature Readings:
        Current temperature : 36.0 deg-C
Sensor A Temperature Readings:
        Current temperature : 41.5 deg-C
        Warning level.......: 69.0 deg-C
        Shutdown level......: 79.0 deg-C

For comparison using a sound meter.

6450 After Firmware Update and Fully Booted

Front of Switch
60db
Back of Switch
61db
At Desk
48db

6610 After Firmware Update 1 PSU and Fully Booted

Front of Switch
60db
Back of Switch
75db
At Desk
56db

Definitely will be doing a fan mod on mine to make it silent.

The popular model is Sunon 40x40x20mm KDE1204PKVX with ~10cfm and about 28dba noise.

Am I not mistaken that the switch is not using a standard pin out and wires will need to be swapped?

Unlike the Aruba S2500 I just did a mod on that had limited space for the fan depth, this switch does not if it came with a 20mm fan you could easily fit a larger 28mm fan.

Here are the included fans in my switch.
Foxconn PIA040H12P

 

[Zervun]

there's something seriously wrong with your metering or your switches, I've never seen them draw more than ~120w even under heavy load (assuming we're talking about 6610's)

Good to know I thought I had read somewhere that they were around 120 on load -

Haven't tested them myself (my UPS isn't hooked up yet) - I also only use a handful of ports on POE. The datasheet from brocade on then had them listed quite a bit higher, I figured that wasn't accurate. It was more to the point that 2x 24's would probably be more juice than 1x 6610, if the scaled power usage is correct in the brocade readme. Was just speculation on my part going by the below numbers that the 24P uses about 80% of the 48P. They don't specify if that its fully catted out ports though.

https://ruckus-www.s3.amazonaws.com/pdf/datasheets/ds-icx-6610.pdf

Models
With 1 Power Supply With 2 Power Supplies
Brocade ICX 6610-24
120 W 140 W
Brocade ICX 6610-48
165 W 185 W
Brocade ICX 6610-24F
125 W 145 W
Brocade ICX 6610-24P
120 W 140 W
Brocade ICX 6610-48P
165 W 185 W

 

[Zervun]

The QSFP+ ports in the back do not require any license, the front SFP+ ports need 2 licences in order to use them with 10G.

Thanks, didn't know that.

 

[Zervun]

No ears thats a major bummer. I will have to find some or make some. I do not like switches on a shelf where they can slide around.

Even with one ear on each of mine (I only had one pair out of 2 switches) they slid around a bit. I ended up putting on some rubber feet on the bottom of 4 corners I had laying around and it has worked fairly decent.

 

[acbaldwi]

Got my 6610 all licensed up and a basic config with 3 vlans on it vlan 1 is wher enormal data flows into the house, vlan 2 is a guest only vlan , vlan 99 is my untrusted vlan for things like iot and such

Router is a ubiquiti usg, DHCP is runnign on this for all 3 vlans
WAP's are 2 ac pro's

ports 1/1/2 and 1/1/4 are poe enabled and the ac pros are on them.

Vlan 2 and vlan 99 do not appear to be getting dhcp from the unifi usg, not sure why this worked on my cisco.

below is my sh vlan, i think i got it right but obv i missed something.... multimode is enabled of course....

SSH@6610>sh vlan
Total PORT-VLAN entries: 3
Maximum PORT-VLAN entries: 64
Legend: [Stk=Stack-Id, S=Slot]
PORT-VLAN 1, Name DEFAULT-VLAN, Priority level0, Spanning tree Off
Untagged Ports: (U1/M1) 1 3 5 7 9 11 13 14 15 16 17 18
Untagged Ports: (U1/M1) 19 20 21 22 23 24 25 26 27 28 29 30
Untagged Ports: (U1/M1) 31 32 33 34 35 36 37 38 39 40 41 42
Untagged Ports: (U1/M1) 43 44 45 46 47 48
Untagged Ports: (U1/M2) 1 2 3 4 5 6 7 8 9 10
Untagged Ports: (U1/M3) 1 2 3 4 5 6 7 8
Tagged Ports: None
Uplink Ports: None
DualMode Ports: (U1/M1) 2 4 6 8 10 12
Mac-Vlan Ports: None
Monitoring: Disabled
PORT-VLAN 2, Name [None], Priority level0, Spanning tree Off
Untagged Ports: None
Tagged Ports: (U1/M1) 2 4 6 8 10 12
Uplink Ports: None
DualMode Ports: None
Mac-Vlan Ports: None
Monitoring: Disabled
PORT-VLAN 99, Name [None], Priority level0, Spanning tree Off
Untagged Ports: None
Tagged Ports: (U1/M1) 2 4 6 8 10 12
Uplink Ports: None
DualMode Ports: None
Mac-Vlan Ports: None
Monitoring: Disabled


As always any help would be greatly appreciated

 

[ViciousXUSMC]

Even with one ear on each of mine (I only had one pair out of 2 switches) they slid around a bit. I ended up putting on some rubber feet on the bottom of 4 corners I had laying around and it has worked fairly decent.

I think I am going to end up doing something like this:
1U Rack Mount 4-Post Shelf Rail for rackmount/tower server - 33.25" deep 695974749392 | eBay

More support than ears only, wont take more than 1U unlike some of the other shelf ideas, and because there is a sidewall area I will run a couple of screws through the sidewall and into the screw holes for the switch where the ears would have mounted to root the switch firmly into place. Can add rubber washers/bushings as well.

My only hesitation is that $30 is still a lot to spend, hope to find something near $15-$20.

 

[Zervun]

I think I am going to end up doing something like this:
1U Rack Mount 4-Post Shelf Rail for rackmount/tower server - 33.25" deep 695974749392 | eBay

More support than ears only, wont take more than 1U unlike some of the other shelf ideas, and because there is a sidewall area I will run a couple of screws through the sidewall and into the screw holes for the switch where the ears would have mounted to root the switch firmly into place. Can add rubber washers/bushings as well.

My only hesitation is that $30 is still a lot to spend, hope to find something near $15-$20.

Ya, I didn't depend on the ears to support them. I have mine on a 1u shelf. The single ear on each one is to just hold it in place a little bit. It slid around still so I put the rubber feet on.

I was going to reach out to a metal smithing place over the next few weeks and see if I can get them to duplicate the ears. Depending on how that works out I could probably hook others up. If it is cost prohibitive I probably won't do it (never went to a metal smithing place before to get something done). Again these are just the short ears at least the ones I got not the RMK-4 rack kit, and I wouldn't trust them to hold the switch on their own.

 

[Hrast]

Vlan 2 and vlan 99 do not appear to be getting dhcp from the unifi usg, not sure why this worked on my cisco.

below is my sh vlan, i think i got it right but obv i missed something.... multimode is enabled of course....

I'm not sure this is your problem, but when I was setting up my IOT vlan, I kept getting the Dual Mode command backwards, so I was adding a tag for the wrong vlan to the port (ie, in your case, I would have been tagging vlan 1, instead of 2 and/or 99).

Here's the vlan bits from my config:

telnet@ICX6430-24 Switch#show running-config vlan
vlan 1 name DEFAULT-VLAN by port
!
vlan 100 by port
!
vlan 200 by port
!
vlan 1010 by port
 tagged ethe 1/1/2 ethe 1/1/7
!
!

telnet@ICX6430-24 Switch#show running-config interface eth 1/1/2
interface ethernet 1/1/2
 port-name From-firewall
 dual-mode
 spanning-tree 802-1w admin-edge-port
!

telnet@ICX6430-24 Switch#show running-config interface eth 1/1/7
interface ethernet 1/1/7
 port-name wireless
 dual-mode
 spanning-tree 802-1w admin-edge-port

Good luck!

 

[acbaldwi]

I'm not sure this is your problem, but when I was setting up my IOT vlan, I kept getting the Dual Mode command backwards, so I was adding a tag for the wrong vlan to the port (ie, in your case, I would have been tagging vlan 1, instead of 2 and/or 99).

Here's the vlan bits from my config:

telnet@ICX6430-24 Switch#show running-config vlan
vlan 1 name DEFAULT-VLAN by port
!
vlan 100 by port
!
vlan 200 by port
!
vlan 1010 by port
 tagged ethe 1/1/2 ethe 1/1/7
!
!

telnet@ICX6430-24 Switch#show running-config interface eth 1/1/2
interface ethernet 1/1/2
 port-name From-firewall
 dual-mode
 spanning-tree 802-1w admin-edge-port
!

telnet@ICX6430-24 Switch#show running-config interface eth 1/1/7
interface ethernet 1/1/7
 port-name wireless
 dual-mode
 spanning-tree 802-1w admin-edge-port

Good luck!

Thanks I think i have mine setup the same way you do here is more info for anyone that may know what ive messed up

SSH@6610>show running-config vlan
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
!
vlan 2 by port
tagged ethe 1/1/2 ethe 1/1/4 ethe 1/1/6 ethe 1/1/8 ethe 1/1/10 ethe 1/1/12 ethe 1/1/41
!
vlan 99 by port
tagged ethe 1/1/2 ethe 1/1/4 ethe 1/1/6 ethe 1/1/8 ethe 1/1/10 ethe 1/1/12 ethe 1/1/41
!
!

SSH@6610>show running-config int e 1/1/41
interface ethernet 1/1/41
dual-mode
!


SSH@6610>show running-config interface eth 1/1/2
interface ethernet 1/1/2
port-name WAP1
dual-mode
inline power
!

SSH@6610>show vlan
Total PORT-VLAN entries: 3
Maximum PORT-VLAN entries: 64

Legend: [Stk=Stack-Id, S=Slot]

PORT-VLAN 1, Name DEFAULT-VLAN, Priority level0, Spanning tree Off
Untagged Ports: (U1/M1) 1 3 5 7 9 11 13 14 15 16 17 18
Untagged Ports: (U1/M1) 19 20 21 22 23 24 25 26 27 28 29 30
Untagged Ports: (U1/M1) 31 32 33 34 35 36 37 38 39 40 42 43
Untagged Ports: (U1/M1) 44 45 46 47 48
Untagged Ports: (U1/M2) 1 2 3 4 5 6 7 8 9 10
Untagged Ports: (U1/M3) 1 2 3 4 5 6 7 8
Tagged Ports: None
Uplink Ports: None
DualMode Ports: (U1/M1) 2 4 6 8 10 12 41
Mac-Vlan Ports: None
Monitoring: Disabled

PORT-VLAN 2, Name [None], Priority level0, Spanning tree Off
Untagged Ports: None
Tagged Ports: (U1/M1) 2 4 6 8 10 12 41
Uplink Ports: None
DualMode Ports: None
Mac-Vlan Ports: None
Monitoring: Disabled

PORT-VLAN 99, Name [None], Priority level0, Spanning tree Off
Untagged Ports: None
Tagged Ports: (U1/M1) 2 4 6 8 10 12 41
Uplink Ports: None
DualMode Ports: None
Mac-Vlan Ports: None
Monitoring: Disabled

SSH@6610>show int e 1/1/41
GigabitEthernet1/1/41 is up, line protocol is up
Port up for 12 hour(s) 10 minute(s) 5 second(s)
Hardware is GigabitEthernet, address is cc4e.24c6.2a74 (bia cc4e.24c6.2a9c)
Configured speed auto, actual 1Gbit, configured duplex fdx, actual fdx
Configured mdi mode AUTO, actual MDI
Member of 3 L2 VLANs, port is dual mode in Vlan 1, port state is FORWARDING
BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
Link Error Dampening is Disabled
STP configured to ON, priority is level0, mac-learning is enabled
Openflow is Disabled, Openflow Hybrid mode is Disabled, Flow Control is config enabled, oper enabled, negotiation disabled
Mirror disabled, Monitor disabled
Mac-notification is disabled
Not member of any active trunks
Not member of any configured trunks
No port name
Inter-Packet Gap (IPG) is 96 bit times
MTU 1500 bytes, encapsulation ethernet
300 second input rate: 7340912 bits/sec, 620 packets/sec, 0.73% utilization
300 second output rate: 72136 bits/sec, 93 packets/sec, 0.00% utilization
4878964832 packets input, 734235379274 bytes, 0 no buffer
Received 250420 broadcasts, 283591 multicasts, 4878430821 unicasts
1 input errors, 1 CRC, 0 frame, 0 ignored
0 runts, 0 giants
1315300962 packets output, 151602772406 bytes, 0 underruns
Transmitted 36745 broadcasts, 306989 multicasts, 1314957228 unicasts
0 output errors, 0 collisions
Relay Agent Information option: Disabled

Egress queues:
Queue counters Queued packets Dropped Packets
0 241550995 1664
1 0 0
2 0 0
3 0 0
4 0 0
5 8143 0
6 0 0
7 0 0

SSH@6610>sh int e 1/1/2
GigabitEthernet1/1/2 is up, line protocol is up
Port up for 29 minute(s) 21 second(s)
Hardware is GigabitEthernet, address is cc4e.24c6.2a74 (bia cc4e.24c6.2a75)
Configured speed auto, actual 1Gbit, configured duplex fdx, actual fdx
Configured mdi mode AUTO, actual MDIX
Member of 3 L2 VLANs, port is dual mode in Vlan 1, port state is FORWARDING
BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
Link Error Dampening is Disabled
STP configured to ON, priority is level0, mac-learning is enabled
Openflow is Disabled, Openflow Hybrid mode is Disabled, Flow Control is config enabled, oper enabled, negotiation disabled
Mirror disabled, Monitor disabled
Mac-notification is disabled
Not member of any active trunks
Not member of any configured trunks
Port name is WAP1
Inter-Packet Gap (IPG) is 96 bit times
MTU 1500 bytes, encapsulation ethernet
300 second input rate: 4080 bits/sec, 1 packets/sec, 0.00% utilization
300 second output rate: 6544 bits/sec, 5 packets/sec, 0.00% utilization
5743229 packets input, 1071336814 bytes, 0 no buffer
Received 163756 broadcasts, 63707 multicasts, 5515766 unicasts
0 input errors, 0 CRC, 0 frame, 0 ignored
0 runts, 0 giants
13538180 packets output, 16277330870 bytes, 0 underruns
Transmitted 165334 broadcasts, 278233 multicasts, 13094613 unicasts
0 output errors, 0 collisions
Relay Agent Information option: Disabled

Egress queues:
Queue counters Queued packets Dropped Packets
0 13537010 771
1 0 0
2 0 0
3 0 0
4 0 0
5 1007 0
6 0 0
7 163 0
SSH@6610>

 

[Hrast]

The VLAN config looks right, however, I have a 6430 (no routing), so my suspicion is that its something to do with your routing configuration.

 

[acbaldwi]

The VLAN config looks right, however, I have a 6430 (no routing), so my suspicion is that its something to do with your routing configuration.

i found it, well i fixed it....

I was uplinking my brocade to my old cisco 2960 while i configured the brocade.... apparently the vlans dont like that.... moved my router over to my brocade and like magic it works.....

Thanks again

 

[dlaffin1012]

Alright I have mixed feelings on stacking and LACP at this point. I see a lot of people who talk about stacking on this thread and homelab stacking. Yes this is not a full homelab and is the main network but until I get to full homelab, I work with this. I want to at least get a 6610 series. At this point do many of you do stacking homelab or just home? It would give me more ports although I would not necessarily use them but it would be nice to have them. The main thing is that pricing doubles. So does power. If I were to stack I would want to LACP any of my 40G connections and 10G. There would be a few 1G ethernet connections on LACP too. Is the stacking and LACP worth the downsides such as cost and power? I would need to make the decision before I purchase equipment as I would need dual port NICs instead of single ports NICs. I know enough to set this up, I just don't know enough if it is worth the benefits for homelab. Although I suppose I could always get a second switch in the future when expanding and just buy the dual ports then. Idk, some please enlighten me more.

I do appreciate the help so far as it has leaned me towards a single switch at this point.

 

[kapone]

Alright I have mixed feelings on stacking and LACP at this point. I see a lot of people who talk about stacking on this thread and homelab stacking. Yes this is not a full homelab and is the main network but until I get to full homelab, I work with this. I want to at least get a 6610 series. At this point do many of you do stacking homelab or just home? It would give me more ports although I would not necessarily use them but it would be nice to have them. The main thing is that pricing doubles. So does power. If I were to stack I would want to LACP any of my 40G connections and 10G. There would be a few 1G ethernet connections on LACP too. Is the stacking and LACP worth the downsides such as cost and power? I would need to make the decision before I purchase equipment as I would need dual port NICs instead of single ports NICs. I know enough to set this up, I just don't know enough if it is worth the benefits for homelab. Although I suppose I could always get a second switch in the future when expanding and just buy the dual ports then. Idk, some please enlighten me more.

I do appreciate the help so far as it has leaned me towards a single switch at this point.

The short answer is...there is no short answer.

You're talking about building redundancy into your switch(es), NICs, servers etc.

The question of whether this is "worth" for a homelab is not the right question, in my imho. The right question is...what ARE you doing with your homelab?? A homelab by definition is a LAB, not a production data center. If you need to experiment with stacking, redundancy etc etc, knock yourself out, otherwise... this is all a moot point?

 

[dlaffin1012]

At this point I don't need to experiment with it but I have VM's and two storage servers that the house and other applications run off of. I am trying to determine if stacking is worth the cost to keep these systems running. They are not for business, so it isn't money lost if they go down.