Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
A

aaroneaton

New Member
Jan 15, 2021
12
0
1
www.rfehosting.com
I have been a long time lurker here for a year or so now. Anyway have had a brocade icx6610 for about a year and a few months, and just barely getting into trying to use the qsfp ports with breakout to 10Gbps ports.

Anyway i am having issues getting them to work. I do not have stacking enabled, I have purchased the below from FS.com
Cisco QSFP-4SFP10G-CU3M 40G QSFP+ Breakout DAC Cable - FS
I was not able to get it working, so emailed them and they said i need a customized one, ordered that, and still no go.

My target end on each port is intel 10Gb nics in dell R620 servers.
reg fs DAC cables work on each server, but am getting no connection lights or anything using the breakout cable and each of its 4 ports.

Any other tips or tricks you can suggest? I have read through just about in this thread about getting breakout ports working, and not found my solution.

Looking forward to your reply.
 
I

itronin

Well-Known Member
Nov 24, 2018
1,353
896
113
Denver, Colorado
aaroneaton said:
I have been a long time lurker here for a year or so now. Anyway have had a brocade icx6610 for about a year and a few months, and just barely getting into trying to use the qsfp ports with breakout to 10Gbps ports.

Anyway i am having issues getting them to work. I do not have stacking enabled, I have purchased the below from FS.com
Cisco QSFP-4SFP10G-CU3M 40G QSFP+ Breakout DAC Cable - FS
I was not able to get it working, so emailed them and they said i need a customized one, ordered that, and still no go.
Click to expand...
please post your config. Yeah, I know you said you have stacking disabled - but worth a look.

For the rear connections:

I'm currently using AOC QSFP to SFP+ breakouts (generic/cisco) and they work great. I've tested generic QSFP to SFP+ DAC - worked fine too.
I've used netapp QSFP to QSFP for stacking on both the 40gbe and breakouts. - worked fine
I've used AOC QSFP to QSFP for switch to host - that also worked fine.

Nothing magical in the cables I've used - in fact they were the cheapest I could find of their respective types - configs were pretty much stock on the 1/2/x side.

when you say reg fs DAC cables work, going from the front sfp+ connections to the servers?

what did show int give you on the breakout ports when you were testing?

for example here's a snippet from my show run on a standalone icx6610 and as you can see nothing configured for my 1/2/x ports and they all work for hosts:

Code: 
!
lag LAG41 dynamic id 41
 ports ethernet 1/2/1 ethernet 1/2/6 
 primary-port 1/2/6
 deploy
!
...
!
interface ethernet 1/1/24
 dual-mode  249
 inline power
!
interface ethernet 1/3/1
 speed-duplex 10G-full
!
2 10gbe in use, 40gbe configured but host is down at the moment. That's with a generic 40gbe to 10gbe AOC breakout.

Code: 
SSH@icx6610-stack#show inter br ethe 1/2/1 to 1/2/10 

Port       Link    State   Dupl Speed Trunk Tag Pvid Pri MAC             Name
1/2/1      Down    None    None None  41    Yes N/A  0   748e.f8dc.ae80                 
1/2/2      Down    None    None None  None  Yes N/A  0   748e.f8dc.ae80                 
1/2/3      Down    None    None None  None  Yes N/A  0   748e.f8dc.ae80                 
1/2/4      Down    None    None None  None  Yes N/A  0   748e.f8dc.ae80                 
1/2/5      Up      Forward Full 10G   None  Yes N/A  0   748e.f8dc.ae80                 
1/2/6      Down    None    None None  41    Yes N/A  0   748e.f8dc.ae80                 
1/2/7      Down    None    None None  None  Yes N/A  0   748e.f8dc.ae80                 
1/2/8      Down    None    None None  None  Yes N/A  0   748e.f8dc.ae80                 
1/2/9      Down    None    None None  None  Yes N/A  0   748e.f8dc.ae80                 
1/2/10     Up      Forward Full 10G   None  Yes N/A  0   748e.f8dc.ae80                 
SSH@icx6610-stack#
 
S

SuperMiguel

New Member
Jun 17, 2021
23
2
3
I have a ICX6610 48p PoE i have it connected to a Tripp Lite SMART1500LCD UPS, and every time the power goes out or even flickers the switch will go off, nothing else that is connected to the UPS turns off, only the switch, i have tried 1 PSU at the time, tried both connected to UPS, took one out, different port of the back of the UPS, and nothing seems to work, always the same problem... Any ideas? Suggestions?
 
N

NablaSquaredG

Bringing 100G switches to homelabs
Aug 17, 2020
1,841
1,217
113
SuperMiguel said:
I have a ICX6610 48p PoE i have it connected to a Tripp Lite SMART1500LCD UPS, and every time the power goes out or even flickers the switch will go off, nothing else that is connected to the UPS turns off, only the switch, i have tried 1 PSU at the time, tried both connected to UPS, took one out, different port of the back of the UPS, and nothing seems to work, always the same problem... Any ideas? Suggestions?
Click to expand...
I have also noticed that the Brocade PSUs are a bit more susceptible to power issues than standard PSUs. Which revision do you have? I will check with ours ASAP

EDIT: According to the datasheet, your Tripp Lite UPS outputs a modified sinewave... That might be the issue.
High end PSUs typically don't like non sinusodial voltage inputs, whereas cheaper PSUs (with just passive PFC for example) can bear with it...
 
Last edited:
S

SuperMiguel

New Member
Jun 17, 2021
23
2
3
NablaSquaredG said:
I have also noticed that the Brocade PSUs are a bit more susceptible to power issues than standard PSUs. Which revision do you have? I will check with ours ASAP

EDIT: According to the datasheet, your Tripp Lite UPS outputs a modified sinewave... That might be the issue.
High end PSUs typically don't like non sinusodial voltage inputs, whereas cheaper PSUs (with just passive PFC for example) can bear with it...
Click to expand...
I have the revision A.

"your Tripp Lite UPS outputs a modified sinewave" is there anything I can do about it? Or basically get another UPS?
 
Last edited:
nickf1227

nickf1227

Active Member
Sep 23, 2015
197
129
43
34
nerdalertdk said:
Hi guys @JoJoMan @nick5768

Have you tried this command ?

Enables support for Power over Ethernet (PoE) legacy power-consuming devices.
Code: 
legacy-inline-power

legacy-inline-power

Enables support for Power over Ethernet (PoE) legacy power-consuming devices.
docs.ruckuswireless.com docs.ruckuswireless.com
Click to expand...
Hello,
Yes. That specific command helps in some specific scenarios, like installing very old IP telephones running very old firmwares. Think Cisco 7940.

The problems I am describing is that under normal circumstances with 802.3af/at compliant devices I've had multiple failures of the POE daughter board on my 7450s and to a lesser extend my 6450s.

The other problem I've noticed is that non-isolated POE devices (which, in fairness, is out of spec) cause havoc on Brocade switches. Cisco and HP Procurve switches handle Pis with non-isolated POE hats without issues, whereas on Brocade if ANY OTHER DEVICES is plugged into a Pi with that hat on, the entire POE daughter board barfs all over itself and stops delivering power to all ports in an infinite loop.

It's not that the Brocade switches aren't any good, it's that they are much more delicate than other enterprise grade switches I have experience with. I've had Cisco 6509s and 4506s running in rooms that are over 100 degrees during summer months for years. I've found HP 2910 and 2920 switches running in ceilings, inside of wooden cabinets with closed doors and no airflow. I have a fleet of Brocade 7450s, some of them are in rooms with dedicated CRAC units and others without, all of them are running behind Eaton 9K or 5KUPSs...but in either case I still see POE module failures.
 
Last edited:
N

noduck

Member
Sep 12, 2020
41
13
8
JoJoMan said:
just tried this, no dice

When the 1 pi shows it, it shows as using 802.3af so shouldn't need legacy power anyway
Click to expand...
JoJoMan said:
I did some more messing around, and I think I have a similar issue to @nickf1227 , I can power 1 pi properly with POE
The other 2 just have some lights come on showing it's getting SOME power, seems like its not enough to boot though as they don't connect to the network, and the ethernet connection/activity lights never come on.

if I disconnect the properly working pi, it seems like it all of a sudden has enough power and one of the 2 not-working pis will start working.

Also, I double checked to make sure I have the latest firmware for everything, I installed the 08030u and updated POE firmware as well, just in case
Click to expand...
I have no issue powering multiple Pis from both a 7250-24p and a c7150-12 (3 each, 6 total). I am not using any POE hat, but external POE adapters:

Amazon.com: PoE Texas GAF-MICROUSB | Gigabit Power Over Ethernet Splitter for 5 Volt USB Power to Tablets, Dropcam, Raspberry Pi and More - Includes Right and Left Angle MicroUSB Adapters : Everything Else

Amazon.com: PoE Texas GAF-MICROUSB | Gigabit Power Over Ethernet Splitter for 5 Volt USB Power to Tablets, Dropcam, Raspberry Pi and More - Includes Right and Left Angle MicroUSB Adapters : Everything Else
www.amazon.com

Amazon.com: PoE Texas GAT-USBC-REV2 | Converts PoE+ to USB Type C Compatible with 12.9” iPad Pro and More | Converts POE to 5v, 9v, 12v, 15v or 20 Volts with Divided Gigabit Data : Electronics

Amazon.com: PoE Texas GAT-USBC-REV2 | Converts PoE+ to USB Type C Compatible with 12.9” iPad Pro and More | Converts POE to 5v, 9v, 12v, 15v or 20 Volts with Divided Gigabit Data : Electronics
www.amazon.com
 
nickf1227

nickf1227

Active Member
Sep 23, 2015
197
129
43
34
The Brocades will power literally as many pis as you can plug into them, but they have to have ISOLATED POE hats. I can't speak for the USB adapter above, but I trust they will probably work fine if that poster is using them. POE Texas is fantastic
 
  • Like
Reactions: JoJoMan
J

JoJoMan

New Member
Jul 19, 2021
10
6
3
nickf1227 said:
The Brocades will power literally as many pis as you can plug into them, but they have to have ISOLATED POE hats. I can't speak for the USB adapter above, but I trust they will probably work fine if that poster is using them. POE Texas is fantastic
Click to expand...
this is probably my issue then, the hat I am using is non-isolated.
I ordered some new isolated hats and will update when those arrive.
 
nickf1227

nickf1227

Active Member
Sep 23, 2015
197
129
43
34
JoJoMan said:
this is probably my issue then, the hat I am using is non-isolated.
I ordered some new isolated hats and will update when those arrive.
Click to expand...
Best of luck, please report back your findings and model you chose.
This model should work fine: Amazon.com: LoveRPi Power-Over-Ethernet (PoE) HAT for Raspberry Pi 4 Model B and Raspberry Pi 3 Model B+ (Professional, Isolated (3KV)) : Electronics

Just for those who are lurking here, "non-isolated POE" is literally not compliant with 802.3 so in Brocade's defense it's not really their bad. It's just disappointing that the other major brands can handle it without issue.
Navigating the IEEE 802.3af Standard for PoE | Power Electronics

Some UniFi switches had a similar problem:
1629775784330.png

The actual standard specifies the following:
1629776015720.png

I'm not sure exactly in the Brocade version of the POE daughter board causes the inability to work properly with the crappy POE hats. Brocade claims that they are IEEE 802.3af/at compliant, which would include the above provision on the PSE side. I'm not an electrical engineer, but the behavior I have had and JoJoMan is having makes me curious...I'm not sure how power feedback from one POE device should have any affect on other POE devices if the switches are following the standard properly.
 
  • Like
Reactions: tommybackeast and JoJoMan
A

aaroneaton

New Member
Jan 15, 2021
12
0
1
www.rfehosting.com
After a whole lot of tests i was able to find out that the ports are bad on the switch itself.. Fun times!


itronin said:
please post your config. Yeah, I know you said you have stacking disabled - but worth a look.

For the rear connections:

I'm currently using AOC QSFP to SFP+ breakouts (generic/cisco) and they work great. I've tested generic QSFP to SFP+ DAC - worked fine too.
I've used netapp QSFP to QSFP for stacking on both the 40gbe and breakouts. - worked fine
I've used AOC QSFP to QSFP for switch to host - that also worked fine.

Nothing magical in the cables I've used - in fact they were the cheapest I could find of their respective types - configs were pretty much stock on the 1/2/x side.

when you say reg fs DAC cables work, going from the front sfp+ connections to the servers?

what did show int give you on the breakout ports when you were testing?

for example here's a snippet from my show run on a standalone icx6610 and as you can see nothing configured for my 1/2/x ports and they all work for hosts:

Code: 
!
lag LAG41 dynamic id 41
ports ethernet 1/2/1 ethernet 1/2/6
primary-port 1/2/6
deploy
!
...
!
interface ethernet 1/1/24
dual-mode  249
inline power
!
interface ethernet 1/3/1
speed-duplex 10G-full
!
2 10gbe in use, 40gbe configured but host is down at the moment. That's with a generic 40gbe to 10gbe AOC breakout.

Code: 
SSH@icx6610-stack#show inter br ethe 1/2/1 to 1/2/10

Port       Link    State   Dupl Speed Trunk Tag Pvid Pri MAC             Name
1/2/1      Down    None    None None  41    Yes N/A  0   748e.f8dc.ae80                
1/2/2      Down    None    None None  None  Yes N/A  0   748e.f8dc.ae80                
1/2/3      Down    None    None None  None  Yes N/A  0   748e.f8dc.ae80                
1/2/4      Down    None    None None  None  Yes N/A  0   748e.f8dc.ae80                
1/2/5      Up      Forward Full 10G   None  Yes N/A  0   748e.f8dc.ae80                
1/2/6      Down    None    None None  41    Yes N/A  0   748e.f8dc.ae80                
1/2/7      Down    None    None None  None  Yes N/A  0   748e.f8dc.ae80                
1/2/8      Down    None    None None  None  Yes N/A  0   748e.f8dc.ae80                
1/2/9      Down    None    None None  None  Yes N/A  0   748e.f8dc.ae80                
1/2/10     Up      Forward Full 10G   None  Yes N/A  0   748e.f8dc.ae80                
SSH@icx6610-stack#
Click to expand...
 
  • Wow
Reactions: itronin
L

LodeRunner

Active Member
Apr 27, 2019
557
237
43
Has anyone seen this boot message before?
Code: 
soc_do_init: soc_do_init: total cpu and arm cosq 0030 unexpected
Getting that on a 7450 after updating to 8090UFI.
 
I

itronin

Well-Known Member
Nov 24, 2018
1,353
896
113
Denver, Colorado
aaroneaton said:
dang! what is another switch that has the dual 40gb Qsfp with breakout comparability? that may be easier to get?
Click to expand...
o_O Another ICX6610

Not sure if you are in the US but there are a ton of listings on the bay - you might also post in the WTB thread and someone may have one for you too? think of the one you have as a parts donor for fan(s) and PSU(S)
 
You must log in or register to reply here.
Top Bottom