Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[kapone]

So to further qualify my original question on connectivity, what should I be using between the rack devices to maximize throughput in a cost-effective manner? If it is reasonably affordable, I would like max speed (40?) between the two 6610s in particular. Then I have my ESXI host and my pfSense. I'm a "buy once, cry once" kinda guy, so I am just trying to future proof this as much as possible. To frame this, if a 12TB HDD is 40% more expensive than the 8TB, I am going with the 8TB. Hopefully that paints a more clear picture of what I am going after here. Thanks in advance for any constructive input.

There's a number of ways to go about this.

- Stack using two of the 40gbps ports on the back and you end up with 80gbps bandwidth between the switches and 32x 10gbps ports for devices.
- Stack using two 4x10g ports on the back and you end up with 80gbps bandwidth between the switches and 2x 40gbps ports and 16x 10gbps ports for devices.
- Stack using all 4 40gbps ports on the back and you end with 160gbps bandwidth between the switches and 16x 10g ports for devices.
- Stack using a single 40gb port....you get the idea.

(I'm not counting the 1gb ports, they are what they are).

 

[StarsAndBars]

Thank you, that does help. In terms of how to physically achieve that, what hardware and cables would I use? How do you prefer to make your rack device connections? I am leaning towards using two of the 40 ports to stack so I maximize future 10gbps for devices in the future. I think 80 is plenty for a home backplane... I don't move THAT much data THAT often. I'm also trying to avoid buying more interconnect gear than I need. I'm currently using 10GbE copper transceivers on my pfSense and ESXI units. Thanks again for your help!

 

[kapone]

Thank you, that does help. In terms of how to physically achieve that, what hardware and cables would I use? How do you prefer to make your rack device connections? I am leaning towards using two of the 40 ports to stack so I maximize future 10gbps for devices in the future. I think 80 is plenty for a home backplane... I don't move THAT much data THAT often. I'm also trying to avoid buying more interconnect gear than I need. I'm currently using 10GbE copper transceivers on my pfSense and ESXI units. Thanks again for your help!

Lots of options...too many to list.

Between the switches...all you need is QSFP cables. They're...dirt cheap.

Between devices and the switch(es) - at 10g - Dirt cheap.

Between devices and the switch(es) - at 1gb - er...

 

[Spearfoot]

Thank you for the follow up. If you were in my shoes, and had all of your gear in a single rack, would you still use one of the breakout cables such as the one you linked? I have a whitebox pfSense firewall that has two SFP ports, and then I have a second (almost identical) whitebox (both are Supermicro E300) that I run ESXI 7 on. If I were to get a second 6610 I could then interconnect both of the 6610s and the two whiteboxes all with a single breakout cable, correct?

Thanks, I am just looking for the most cost-effective (but also high-performing) method to link up this gear. I know there is no real benefit in even 10G for the pfSense box, considering my ISP isn't even gigabit yet, but I would think 10G (or faster) would be the way to go between my 2 6610s and the ESXI server.

You must have a lot of gear if you need to stack two of these 48-port switches!

The man to ask about stacking is @fohdeesha, but as I understand it, of the 4 x 40G ports on the rear of these switches: 2 can only be used at 40G, and two can only be used with breakout cables; all can be used for stacking or for data (see post #4 in this thread). The breakout ports give you 4 SFP+ connections that you can use pretty much however you want.

You could stack your two 6610s using the 40G ports and still use two breakout cables on each switch. That's 8 SFP+ breakout connections per switch, and you'd still have the 8 SFP ports on the front panel of each switch, too. Wowza!

So we have lots of options with these 6610s -- they're beasts! -- and there are quite a few ways you can skin the cat:

• If both of your servers have 2 SFP ports, you could use a single breakout cable to connect them to one switch.
• Using a single breakout cable, you may be able to use one SFP connection to stack the two switches and use the other 3 SFP connections for your two servers (assuming one of them only has a single SFP port). This would only give you a 10G stack connection; you'd be better off using the 40G ports for stacking, IHMO. I think this is what you have in mind?
• You could connect your ESXi server with a cheap 40G card like this Mellanox and use either the front panel SFP ports or a breakout cable for your pfSense and other servers. This would still leave a 40G port free for stacking two switches.
• ...many other options I've missed...

I only use a single switch, and I used breakout cables instead of the front panel ports because it makes my rack less cluttered.

What it boils down to is that you might not need a breakout cable at all, depending on how you want to set up your systems.

 

[EngineerNate]

Thank you. Is there a SKU or part number on these PSUs? Does the Revision apply to the chassis itself, or the PSU?

RPS15-E and RPS15-I for non-PoE (E = exhaust out the back, I = intake at the back)

RPS16-E and RPS16-I for PoE.

Revisions are just the the PSUs. They're interchangeable and you can mix revisions, but fan profiles will get worse if you have any rev As in the mix.

 

[StarsAndBars]

RPS15-E and RPS15-I for non-PoE (E = exhaust out the back, I = intake at the back)

RPS16-E and RPS16-I for PoE.

Revisions are just the the PSUs. They're interchangeable and you can mix revisions, but fan profiles will get worse if you have any rev As in the mix.

Thank you, exactly the concise info I needed. Appreciate it!

 

[StarsAndBars]

You must have a lot of gear if you need to stack two of these 48-port switches!

The man to ask about stacking is @fohdeesha, but as I understand it, of the 4 x 40G ports on the rear of these switches: 2 can only be used at 40G, and two can only be used with breakout cables; all can be used for stacking or for data (see post #4 in this thread). The breakout ports give you 4 SFP+ connections that you can use pretty much however you want.

You could stack your two 6610s using the 40G ports and still use two breakout cables on each switch. That's 8 SFP+ breakout connections per switch, and you'd still have the 8 SFP ports on the front panel of each switch, too. Wowza!

So we have lots of options with these 6610s -- they're beasts! -- and there are quite a few ways you can skin the cat:

• If both of your servers have 2 SFP ports, you could use a single breakout cable to connect them to one switch.
• Using a single breakout cable, you may be able to use one SFP connection to stack the two switches and use the other 3 SFP connections for your two servers (assuming one of them only has a single SFP port). This would only give you a 10G stack connection; you'd be better off using the 40G ports for stacking, IHMO. I think this is what you have in mind?
• You could connect your ESXi server with a cheap 40G card like this Mellanox and use either the front panel SFP ports or a breakout cable for your pfSense and other servers. This would still leave a 40G port free for stacking two switches.
• ...many other options I've missed...

I only use a single switch, and I used breakout cables instead of the front panel ports because it makes my rack less cluttered.

What it boils down to is that you might not need a breakout cable at all, depending on how you want to set up your systems.

Remember there is a big difference between device 49 and device 96. I just maxed out my old 48 port switch, and need just a few more ports, particularly PoE. I didn't see the value in buying a 24 port switch as a second unit - just to be safe.

So if I were to buy a total of 4 fiber transceivers to link the two 6610s on their 40G ports, what should I use? Do you have a link? Would the Amazon Basics fiber cables get the job done?

 

[StarsAndBars]

As far as a breakout cable goes to connect my ESXI and pfSense boxes (each one has 2 SFP ports), would this work?

40Gb QSFP+ to 4x 10Gb SFP+ DAC Breakout Cable, 1.5 Meters 1.5M, 30 AWG | eBay

Ideal Applications for SIS's QSFP+ to 4 SFP+ Passive Copper Assemblies. Key Features of SIS's QSFP+ to 4 SFP+ Passive Copper Assemblies. High-density QSFP+ connector. SFF-8431 Compliant SFP+ connector.

www.ebay.com

I found this one as well. I assume Cisco cables are compatible with these Brocade 6610s? Would the fiber used here be "better" considering it isn't copper?

*New*Cisco QSFP-4X10G-AOC3M 40GBase QSFP to 4 SFP+ Active Optical Breakout Cable | eBay

Find many great new & used options and get the best deals for *New*Cisco QSFP-4X10G-AOC3M 40GBase QSFP to 4 SFP+ Active Optical Breakout Cable at the best online prices at eBay! Free shipping for many products!

www.ebay.com

 

[Spearfoot]

Remember there is a big difference between device 49 and device 96. I just maxed out my old 48 port switch, and need just a few more ports, particularly PoE. I didn't see the value in buying a 24 port switch as a second unit - just to be safe.

So if I were to buy a total of 4 fiber transceivers to link the two 6610s on their 40G ports, what should I use? Do you have a link? Would the Amazon Basics fiber cables get the job done?

Your best bet for stacking the switches would be to use a short QSFP DAC/Twinax cable. These can be had for ~$25 and up on eBay. Here's an example:

Cisco 40GBASE-CR4 Passive Copper Cable - Twinaxial QSFP+ 3 m orange for P/N: QSFP-40G-SR4, QSFP-40G-SR4= for sale online | eBay

Find many great new & used options and get the best deals for Cisco 40GBASE-CR4 Passive Copper Cable - Twinaxial QSFP+ 3 m orange for P/N: QSFP-40G-SR4, QSFP-40G-SR4= at the best online prices at eBay! Free shipping for many products!

www.ebay.com

 

[Spearfoot]

As far as a breakout cable goes to connect my ESXI and pfSense boxes (each one has 2 SFP ports), would this work?

40Gb QSFP+ to 4x 10Gb SFP+ DAC Breakout Cable, 1.5 Meters 1.5M, 30 AWG | eBay

Ideal Applications for SIS's QSFP+ to 4 SFP+ Passive Copper Assemblies. Key Features of SIS's QSFP+ to 4 SFP+ Passive Copper Assemblies. High-density QSFP+ connector. SFF-8431 Compliant SFP+ connector.

www.ebay.com

I found this one as well. I assume Cisco cables are compatible with these Brocade 6610s? Would the fiber used here be "better" considering it isn't copper?

*New*Cisco QSFP-4X10G-AOC3M 40GBase QSFP to 4 SFP+ Active Optical Breakout Cable | eBay

Find many great new & used options and get the best deals for *New*Cisco QSFP-4X10G-AOC3M 40GBase QSFP to 4 SFP+ Active Optical Breakout Cable at the best online prices at eBay! Free shipping for many products!

www.ebay.com

Both of those should work. Bear in mind that fiber requires active transceivers and will draw more power and generate more heat than a DAC cable. I use DAC breakout cables on my switch.

 

[EngineerNate]

I'm also curious which cheap breakout cable works best with these switches, I want to trial it in the 7450 QSFP+ modules but if it doesn't work I want one that works with the 6610 so I can resell it haha.

 

[ViciousXUSMC]

Just a friendly bump (this thread moves fast) anybody know a potential solution to my PoE detection problem on the ICX 6450?
A PoE spliter I just bought works fine on a lesser switch, but refuses to power up on the 6450.

I tried all the PoE commands I can find and have not had luck forcing it to deliver power to the device.

I do have a R710 access point working on the switch so I know the PoE does work.

Edit: got a new PoE splitter today and it didn't work either, in effort to troublshoot I moved to different ports. I also tried some of the same commands as before (decouple datalink, since I had no link lights) sure enough when I moved over ot port 8 it worked! Then I went back to 23, 27, tried 32, etc. None of them worked.

That made me believe I had bad ports, I recalled this switch having everything working as I remember flashing the firmware to all the PoE modules when I got it. But I looked up the commands and instead of "show inline power" I tried "show inline power detail"

Sure enough I was greeted with this:

Power Supply Data On stack 1:                                                                                                 
++++++++++++++++++                                                                                                             
                                                                                                                              
                                                                                                                              
                                                                                                                              
Power Supply Data:                                                                                                             
++++++++++++++++++                                                                                                             
                                                                                                                              
Power Supply #1:                                                                                                               
        Max Curr:       13.7 Amps                                                                                             
        Voltage:        54.0 Volts                                                                                             
        Capacity:       740 Watts                                                                                             
power supply 2 is not present                                                                                                 
power supply 3 is not present                                                                                                 
                                                                                                                              
                                                                                                                              
POE Details Info. On Stack 1 :                                                                                                 
                                                                                                                              
                                                                                                                              
General PoE Data:                                                                                                             
+++++++++++++++++                                                                                                             
                                                                                                                              
Firmware                                                                                                                       
Version                                                                                                                       
----------------                                                                                                               
02.1.0 Build 004                                                                                                               
                                                                                                                              
                                                                                                                              
                                                                                                                              
Cumulative Port State Data:                                                                                                   
+++++++++++++++++++++++++++                                                                                                   
                                                                                                                              
#Ports    #Ports     #Ports   #Ports    #Ports       #Ports     #Ports                                                         
Admin-On  Admin-Off  Oper-On  Oper-Off  Off-Denied   Off-No-PD  Off-Fault                                                     
-------------------------------------------------------------------------                                                     
2         46         2        46        0            0          0

I am guessing that means I have bad PoE modules.
Not the end of the world I suppose, but how would I verify that's the case and they cant be brought to life?

Also its 48Ports and its 3 modules. 48/3 is not an even number so just how many ports of PoE do I get with just one working power supply? I think even port 1/1/13 did not work.

 

[noacess]

I'm considering picking up either an ICX6650 or a Arista DCS-7050Q for a mix of QSFP+ and SFP+. Anyone know how the fan noise compares between these? Power consumption seems like it would be similar. Thanks!

 

[kapone]

Just a friendly bump (this thread moves fast) anybody know a potential solution to my PoE detection problem on the ICX 6450?
A PoE spliter I just bought works fine on a lesser switch, but refuses to power up on the 6450.

I tried all the PoE commands I can find and have not had luck forcing it to deliver power to the device.

I do have a R710 access point working on the switch so I know the PoE does work.

Edit: got a new PoE splitter today and it didn't work either, in effort to troublshoot I moved to different ports. I also tried some of the same commands as before (decouple datalink, since I had no link lights) sure enough when I moved over ot port 8 it worked! Then I went back to 23, 27, tried 32, etc. None of them worked.

That made me believe I had bad ports, I recalled this switch having everything working as I remember flashing the firmware to all the PoE modules when I got it. But I looked up the commands and instead of "show inline power" I tried "show inline power detail"

Sure enough I was greeted with this:

Power Supply Data On stack 1:                                                                                                
++++++++++++++++++                                                                                                            
                                                                                                                             
                                                                                                                             
                                                                                                                             
Power Supply Data:                                                                                                            
++++++++++++++++++                                                                                                            
                                                                                                                             
Power Supply #1:                                                                                                              
        Max Curr:       13.7 Amps                                                                                            
        Voltage:        54.0 Volts                                                                                            
        Capacity:       740 Watts                                                                                            
power supply 2 is not present                                                                                                
power supply 3 is not present                                                                                                
                                                                                                                             
                                                                                                                             
POE Details Info. On Stack 1 :                                                                                                
                                                                                                                             
                                                                                                                             
General PoE Data:                                                                                                            
+++++++++++++++++                                                                                                            
                                                                                                                             
Firmware                                                                                                                      
Version                                                                                                                      
----------------                                                                                                              
02.1.0 Build 004                                                                                                              
                                                                                                                             
                                                                                                                             
                                                                                                                             
Cumulative Port State Data:                                                                                                  
+++++++++++++++++++++++++++                                                                                                  
                                                                                                                             
#Ports    #Ports     #Ports   #Ports    #Ports       #Ports     #Ports                                                        
Admin-On  Admin-Off  Oper-On  Oper-Off  Off-Denied   Off-No-PD  Off-Fault                                                    
-------------------------------------------------------------------------                                                    
2         46         2        46        0            0          0

I am guessing that means I have bad PoE modules.
Not the end of the world I suppose, but how would I verify that's the case and they cant be brought to life?

Also its 48Ports and its 3 modules. 48/3 is not an even number so just how many ports of PoE do I get with just one working power supply? I think even port 1/1/13 did not work.

The number of power supplies does not affect the number of ports that you can get POE on. One PSU can power all.

It seems likely that you have a bad POE board. They are very hard to find by themselves.

 

[fohdeesha]

special things made public when we hit page 200

 

[hmw]

Give us a hint

 

[noacess]

special things made public when we hit page 200

Anything that might impact buying decisions?

 

[fohdeesha]

Anything that might impact buying decisions?

probably not, just reduce my time having to reply to PMs by 90000 hours

 

[EngineerNate]

I can help us hit 200.

Still curious which DAC breakout cable is favored. To make this post more on-topic.

 

[klui]

special things made public when we hit page 200

I really want that fan mod for the ICX6600... : ) But just hoping your health troubles are over.