Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

crackelf

Member
Apr 11, 2021
74
6
8
that's even weirder, it's a stretch but what fw are you on? 8095? try going down to 8090mc (its in the website guide package)

also paste me the output of "show run" and "show mod", there might be some stacking port definitions you can try to remove. lastly maybe try forcing the port speed if it'll let you, something like int e 1/3/1 then type speed and hit tab a bunch to see the options
Will try the fw downgrade - give me a few moments to run through it. This whole thing is massively weird to me too. I've tried slamming through the speed-duplex menu before
Code:
speed-duplex 40g-full
This CLI operation is not applicable to port 1/4/1
show run
Code:
Current configuration:
!
ver 08.0.95fT213
!
stack unit 1
  module 1 icx7450-48p-poe-management-module
  module 2 icx7400-xgf-4port-40g-module
  module 3 icx7400-qsfp-1port-40g-module
  module 4 icx7400-qsfp-1port-40g-module
stack disable
!
!
!
!
!
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
!
!
!
!
!
!
!                                                                
!
!
!
!
!
!
optical-monitor
optical-monitor non-ruckus-optic-enable
aaa authentication web-server default local
aaa authentication login default local
chassis fanless all
enable aaa console
ip dhcp-client disable
!
no telnet server
username root password .....
!
!
!
!
no web-management http
!
manager disable                                                  
!
!
manager port-list 987
!
!
!
!
!
!
!
!
!
interface ethernet 1/2/1
speed-duplex 10G-full
!
interface ethernet 1/2/2
speed-duplex 10G-full
!
interface ethernet 1/2/3
speed-duplex 10G-full
!
interface ethernet 1/2/4
no optical-monitor                                              
speed-duplex 10G-full
!
interface ve 1
ip address [redacted let me know if you actually need this]
no ip dhcp-client enable
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
end
show mod
Code:
       Module                                         Status Ports Starting MAC 
U1:M1  ICX7450-48P POE 48-port Management Module        OK     48   cc4e.2488.3380
U1:M2  ICX7400-4X10GF 4-port 40G Module                 OK     4    cc4e.2488.3380
U1:M3  ICX7400-1X40GQ 1-port 40G Module                 OK     1    cc4e.2488.3380
U1:M4  ICX7400-1X40GQ 1-port 40G Module                 OK     1    cc4e.2488.3380
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,788
3,167
113
33
fohdeesha.com
hmm maybe try removing the "stack disable" option, (either "no stack disable" or "stack enable" can't remember which) then rebooting. then if you can paste the output of (assuming 1/3/1 is actually plugged into a card):

show int e 1/3/1
show media ethernet 1/3/1
show optic 1/3/1
 

crackelf

Member
Apr 11, 2021
74
6
8
hmm maybe try removing the "stack disable" option, (either "no stack disable" or "stack enable" can't remember which) then rebooting. then if you can paste the output of (assuming 1/3/1 is actually plugged into a card):

show int e 1/3/1
show media ethernet 1/3/1
show optic 1/3/1
Writing this down as I go here
stack enable
Code:
Error! stack enable fails due to no stack port. Please configure stack-port/trunk on unit 1. (default: stack-port 1/3/1 stack-port 1/4/1)
stack-port 1/2/3
Code:
T=27m26.1: Module 2 was inited with MACsec. Its ports cannot be stack-ports without a reload.
Reload required to form a stack with Module 2 ports. Please write memory and then reload or power cycle.
write mem
reload
show run
Code:
stack unit 1
  module 1 icx7450-48p-poe-management-module
  module 2 icx7400-xgf-4port-40g-module
  module 3 icx7400-qsfp-1port-40g-module
  module 4 icx7400-qsfp-1port-40g-module
  stack-port 1/2/3
stack disable
stack enable
Code:
Success - Enable stacking. This unit actively participates in stacking
Stacking is enabled. Optical Monitoring is not available for the port(s) 1/2/3
show run
Code:
stack unit 1
  module 1 icx7450-48p-poe-management-module
  module 2 icx7400-xgf-4port-40g-module
  module 3 icx7400-qsfp-1port-40g-module
  module 4 icx7400-qsfp-1port-40g-module
  stack-port 1/2/3
stack enable
write mem
reload
sh int e 1/3/1
Code:
  Port down for 7 minute(s) 49 second(s) 
  Hardware is 40GigabitEthernet, address is cc4e.2488.3380 (bia cc4e.2488.33b5)
  Interface type is 40Gig Fiber
  Configured speed 40Gbit, actual unknown, configured duplex fdx, actual unknown
  Configured mdi mode AUTO, actual unknown
  Untagged member of L2 VLAN 1, port state is BLOCKING
  BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
  Link Error Dampening is Disabled
  STP configured to ON, priority is level0, mac-learning is enabled
  MACsec is Disabled
  Openflow is Disabled, Openflow Hybrid mode is Disabled,  Flow Control is config enabled, oper enabled, negotiation disabled
  Mirror disabled, Monitor disabled
  Mac-notification is disabled
  VLAN-Mapping is disabled
  Not member of any active trunks
  Not member of any configured trunks
  No port name
  IPG MII 96 bits-time, IPG GMII 96 bits-time
  MTU 1500 bytes, encapsulation ethernet
  MMU Mode is Store-and-forward
  300 second input rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
  300 second output rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
  0 packets input, 0 bytes, 0 no buffer                           
  Received 0 broadcasts, 0 multicasts, 0 unicasts
  0 input errors, 0 CRC, 0 frame, 0 ignored
  0 runts, 0 giants
  0 packets output, 0 bytes, 0 underruns
  Transmitted 0 broadcasts, 0 multicasts, 0 unicasts
  0 output errors, 0 collisions
  Relay Agent Information option: Disabled
  Protected: No
  MAC Port Security: Disabled
sh med e 1/3/1
Code:
Port   1/3/1: Type  : 40GE-LR4   2km (QSFP+ LC)
         Vendor: KAIAM CORP           Version: 1A
         Part# : XQX2502              Serial#: KD60315209
show optic 1/3/1 returns nothing oddly enough
 

crackelf

Member
Apr 11, 2021
74
6
8
that's even weirder, it's a stretch but what fw are you on? 8095? try going down to 8090mc (its in the website guide package)
I haven't done the "Load The Latest UFI Image" step yet. Do I need to? I don't want to brick this if copy tftp flash 192.168.1.8 ICX7xxx/SPR08095fufi.bin primary conflicts with 8090mc.
show run
Code:
Current configuration:
!
ver 08.0.90mcT213
!
stack unit 1
  module 1 icx7450-48p-poe-management-module
  module 2 icx7400-xgf-4port-40g-module
  module 3 icx7400-qsfp-1port-40g-module
  module 4 icx7400-qsfp-1port-40g-module
  stack-port 1/2/3
sh med e 1/3/1
Code:
Port   1/3/1: Type  : 40GE-LR4   2km (QSFP+ LC)
         Vendor: KAIAM CORP           Version: 1A
         Part# : XQX2502              Serial#: KD60315209
ICX7450-48P Router(config-if-e40000-1/3/1)#show int e 1/3/1
Code:
40GigabitEthernet1/3/1 is down, line protocol is down
  Port down for 9 minute(s) 46 second(s)
  Hardware is 40GigabitEthernet, address is cc4e.2488.3380 (bia cc4e.2488.33b5)
  Interface type is 40Gig Fiber
  Configured speed 40Gbit, actual unknown, configured duplex fdx, actual unknown
  Configured mdi mode AUTO, actual unknown
  Untagged member of L2 VLAN 1, port state is BLOCKING
  BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
[/ICODE]
 

AgentXXL

New Member
Apr 23, 2020
23
5
3
For the 'who the heck cares' crowd: my inability to find an actual Supermicro Aquantia AQS-107-B0C2-CX SFP module led me to purchase a Mikrotik CRS305 and S+RJ10 module. Alas that still seems to have issues with my onboard Aquantia 5Gbps NIC - the AQC111C. Asus support has been useless in my quest to find a firmware updater that works for my onboard NIC. They instead want me to RMA my motherboard. I'm more than certain they'll just say they find nothing wrong and send it back.

So, instead of banging my head against my lack of knowledge, I decided to throw more money at the issue. Anyone using the SFP+ ports on the 6610 with an Intel X550 NIC? As I'm going to follow-through and proceed with the RMA, I needed an option to keep my system running. When I originally built this CoreX system, I wanted to go with the Asus WS x299 SAGE/10 motherboard - this one has dual onboard 10Gbps NICs based on the Intel X550 controller. It goes for about $1K new here in Canada, but there were none to be found so I went with the Asus Prime x299 Deluxe II with the now known to be buggy Aquantia.

Yesterday I found someone selling a used WS x299 SAGE/10 on eBay and took the plunge. I did do some research on the Intel X550 and it seems to be pretty well supported, but there are always going to be setups that have issues. Hopefully those 10G ports will work well with the ICX6610. I'm holding onto the Mikrotik so I'll still have that if the 6610 doesn't like the X550. The replacement motherboard won't be here for at least 10 days so in the meantime I'll continue fighting with the Aquantia issues.

Sigh...
 
  • Like
Reactions: Sprint8

AgentXXL

New Member
Apr 23, 2020
23
5
3
The X550 supports 10Gb so there’s no reason it shouldn’t work with the 6610.
Yes, that's why I'm more hopeful it'll work. One of my other systems has an Aquantia 10Gbps NIC onboard and the 6610 has no discernible issues with it. I just didn't find anyone posting about issues between the Intel X550 and the 6610 but thought I'd ask.

EDIT: Are there any recommended sites/programs for 'free' learning more about networking? Google/DuckDuckGo list many, but was hoping to find something that uses the Brocade switches for their lessons.

EDIT2: This one? Seems to be SAN but probably contains some network training. Brocade Education
 
Last edited:

crackelf

Member
Apr 11, 2021
74
6
8
hmm maybe try removing the "stack disable" option, (either "no stack disable" or "stack enable" can't remember which) then rebooting. then if you can paste the output of (assuming 1/3/1 is actually plugged into a card):

show int e 1/3/1
show media ethernet 1/3/1
show optic 1/3/1
Update: these Connectx-4 LX are running @1Gbps when connected to each other, and I can't get them to connect even directly with anything higher than that. My advice to the thread: stay very far away from Mellanox if you value your time.
 
Last edited:

simbo

New Member
Feb 24, 2022
10
2
3
Got a small problem where I can't get the 10GB breakout interfaces to come up. I'm trying to brake out the 40GB to 10GB LCs using some FS.com gear. I've followed the setup guide for removing the ports from the stack.

To get the 10G LCs on the front of the rack, I've purchased the following:
- QSFP-SR4-40G Brocade 40G-QSFP-SR4 Compatible 40GBASE-SR4 QSFP+ 850nm 150m DOM MTP/MPO MMF Optical Transceiver Module
- 12FMTPOM4 1m (3ft) MTP®-12 (Female) to MTP®-12 (Female) OM4 Multimode Elite Trunk Cable, 12 Fibers, Type B, Plenum (OFNP), Magenta
-
FHD-1MTP4LCDOM4 FHD MTP®-8 Cassette, 8 Fibers OM4 Multimode, Universal Polarity, MTP® to 4 x LC Duplex (Aqua), 0.35dB max

I'm using FS.com 10GB 850nm SR transceiver's on the other end with out any problems over OM4 cables.


Code:
SSH@sw-core(config)#show media validation


Port       Supported Vendor               Type
----------------------------------------------------------------------
1/2/2      Yes       BROCADE              40G QSFP-SR4
1/2/3      Yes       BROCADE              40G QSFP-SR4
1/2/4      Yes       BROCADE              40G QSFP-SR4
1/2/5      Yes       BROCADE              40G QSFP-SR4
1/2/7      Yes       BROCADE              40G QSFP-SR4
1/2/8      Yes       BROCADE              40G QSFP-SR4
1/2/9      Yes       BROCADE              40G QSFP-SR4
1/2/10     Yes       BROCADE              40G QSFP-SR4
1/3/3      Yes       FS                    Type  : 10GE  Passive Twinax  1m (SFP +) (Not supported)
1/3/4      Yes       FS                    Type  : 10GE  Passive Twinax  1m (SFP +) (Not supported)
1/3/5      Yes       FS                    Type  : 10GE  Passive Twinax  5m (SFP +) (Not supported)
1/3/6      Yes       FS                    Type  : 10GE  Passive Twinax  5m (SFP +) (Not supported)
1/3/7      Yes       OEM                   Type  : 10GE SR 300m (SFP +)

1/3/7 is an optic chucked into the front port used to test the LC cables. The other 1/3's are LAGs go to other switchgear.


Code:
SSH@sw-core(config)#show conf
!
Startup-config data location is flash memory
!
Startup configuration:
!
ver 08.0.30uT7f3
!
stack unit 1
  module 1 icx6610-48p-poe-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
stack disable

FWIW, I set the 1/2/2 to 1/2/5 and 1/2/7 to 1/2/10 to speed-duplex 10G-full also left blank.

Here's the interface summary

Code:
1/2/1      Down    None    None None  None  Yes 1    0   748e.f8fe.c148
1/2/2      Down    None    None None  None  No  1    0   748e.f8fe.c148
1/2/3      Down    None    None None  None  Yes 1    0   748e.f8fe.c148
1/2/4      Down    None    None None  None  No  1    0   748e.f8fe.c148
1/2/5      Down    None    None None  None  No  1    0   748e.f8fe.c148
1/2/6      Down    None    None None  None  No  1    0   748e.f8fe.c148
1/2/7      Down    None    None None  None  Yes 1    0   748e.f8fe.c148  SW-XXX1
1/2/8      Down    None    None None  None  Yes 1    0   748e.f8fe.c148  SW-XXX2
1/2/9      Down    None    None None  None  Yes 1    0   748e.f8fe.c148  SW-XXX3
1/2/10     Down    None    None None  None  Yes 1    0   748e.f8fe.c148
1/3/1      Down    None    None None  None  Yes 1    0   748e.f8fe.c148
1/3/2      Down    None    None None  None  Yes 1    0   748e.f8fe.c148
1/3/3      Up      Forward Full 10G   3     Yes 1    0   748e.f8fe.c148  SW-DXX-1
1/3/4      Up      Forward Full 10G   3     Yes 1    0   748e.f8fe.c148  SW-DXX-2
1/3/5      Up      Forward Full 10G   2     Yes 1    0   748e.f8fe.c148  SW-RXX-1
1/3/6      Up      Forward Full 10G   2     Yes 1    0   748e.f8fe.c148  SW-RXX-2
1/3/7      Down    None    None None  None  Yes 1    0   748e.f8fe.c148
1/3/8      Down    None    None None  None  Yes 1    0   748e.f8fe.c148
Here's the interface that has a plugged in LC cable:
Code:
SSH@sw-core(config)#show int ethe 1/2/7
  10GigabitEthernet 1/2/7 is down, line protocol is down
  Port down for 2 hour(s) 25 minute(s) 13 second(s)
  Hardware is   10GigabitEthernet , address is 748e.f8fe.c148 (bia 748e.f8fe.c17f)
  Configured speed 10Gbit, actual unknown, configured duplex fdx, actual unknown
  Configured mdi mode AUTO, actual unknown
  Member of 11 L2 VLANs, port is dual mode in Vlan 1, port state is BLOCKING
  BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
  Link Error Dampening is Disabled
  STP configured to ON, priority is level0, mac-learning is enabled
  Openflow is Disabled, Openflow Hybrid mode is Disabled,  Flow Control is config enabled, oper enabled, negotiation disabled
  Mirror disabled, Monitor disabled
  Mac-notification is disabled
  Not member of any active trunks
  Not member of any configured trunks
  Port name is SW-XXX1
  MTU 10200 bytes, encapsulation ethernet
  300 second input rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
  300 second output rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
  54035 packets input, 15271970 bytes, 0 no buffer
  Received 2649 broadcasts, 3205 multicasts, 48181 unicasts
  0 input errors, 0 CRC, 0 frame, 0 ignored
  0 runts, 0 giants
  129068 packets output, 41755954 bytes, 0 underruns
  Transmitted 11005 broadcasts, 55975 multicasts, 62088 unicasts
  0 output errors, 0 collisions
  Relay Agent Information option: Disabled

Egress queues:
Queue counters    Queued packets    Dropped Packets
    0              115303                   0
    1                   0                   0
    2                   0                   0
    3                   0                   0
    4                   0                   0
    5                   1                   0
    6                   0                   0
Code:
SSH@sw-core>show ver
  Copyright (c) 1996-2016 Brocade Communications Systems, Inc. All rights reserved.
    UNIT 1: compiled on Apr 23 2020 at 13:17:12 labeled as FCXR08030u
                (10545591 bytes) from Primary FCXR08030u.bin
        SW: Version 08.0.30uT7f3
  Boot-Monitor Image size = 370695, Version:10.1.00T7f5 (grz10100)
  HW: Stackable ICX6610-48-HPOE
==========================================================================
UNIT 1: SL 1: ICX6610-48P POE 48-port Management Module
         Serial  #: 2ax5o2jk68e
         License: ICX6610_ADV_ROUTER_SOFT_PACKAGE   (LID: H4CKTH3PLN8)
         P-ENGINE  0: type E02B, rev 01
         P-ENGINE  1: type E02B, rev 01
==========================================================================
UNIT 1: SL 2: ICX6610-QSFP 10-port 160G Module
==========================================================================
UNIT 1: SL 3: ICX6610-8-port Dual Mode(SFP/SFP+) Module
==========================================================================
  800 MHz Power PC processor 8544E (version 0021/0023) 400 MHz bus
65536 KB flash memory
  512 MB DRAM
STACKID 1  system uptime is 5 day(s) 1 hour(s) 38 minute(s) 27 second(s)
The system started at 12:06:39 GMT+10 Sun Feb 20 2022

The system : started=cold start
I've double checked that the cables are plugged in. I have 2 x of the above kit and I get the same behaviour on each 40GQSFP, MTP cable and breakout cassette.

Any ideas on what to try next?
OK.....I reset the speed-duplex to auto and did a cold boot and the interfaces came up as 10G; i.e. working.
 

dwalme

New Member
Jan 16, 2018
5
0
1
45
I just picked up a secondhand 7250-48p with the help of the information in this thread. Thank you.

I am having an issue that POE does not work on some ports and I'm trying to figure out if there is a hardware issue or a software issue.

I followed the guide to get the switch updated and everything looked as I expected. Then I updated to the latest version 9.0.10a.

When I run show inline power detail debug-info I see (6) modules with HW version V1R3. However one module shows Vm2vErr as status and five show VOP-Sev1 as status. Only one is in Good status.

EDIT: I confirmed that the first 8 ports that are in Vm2vErr are the ports that are not powering POE devices.

Does anyone have any reference to these status codes and/or knows how to troubleshoot this? Google search turns up almost nothing.

FastIron Command Reference Guide has this information but I am not sure what causes these statuses or how to correct them.

The status of the device:
• Failed: No (bad or unreachable) PoE device found (bootup time)
• Good: OK - Expected PoE device Found (Zone1, no VOP)
• Revived: Device is currently refreshed
• n/a: Reserved
• Lost: Device lost or different from expected (while in operation)
• VOP-Sev1: Device error 1 (Zone2, VOP severity 1)
• VOP-Sev2: Device error 1 (Zone2, VOP severity 2)
• VOP-Sev3: Device error 1 (Zone2, VOP severity 3)
• VmErr: Device Vmain Error
• Vm2vErr: Device Vmain < System AVG Vmain by 2v or more
• Recovered: OK - Expected PoE device Found (Zone3, recovered from VOP)


Full output of show inline power detail debug-info:
Power Supply Data On unit 1:
++++++++++++++++++



Power Supply Data:
++++++++++++++++++

Power Supply #1:
Max Curr: 13.7 Amps
Voltage: 54.0 Volts
Capacity: 740 Watts
PoePower: 740 Watts
power supply 2 is not present
power supply 3 is not present


POE Details Info. On Unit 1 :


General PoE Data:
+++++++++++++++++

Firmware
Version
----------------
02.1.8 Build 004

Hardware
Version
----------------
V1R3

Device HW version : 0:V1R3 1:V1R3 2:V1R3 3:V1R3 4:V1R3 5:V1R3
Device Temperature(deg-C) : 0:42 1:40 2:40 3:42 4:37 5:39
Device Status : 0:Vm2vErr 1:VOP-Sev1 2:VOP-Sev1 3:VOP-Sev1 4:Good 5:VOP-Sev1



Cumulative Port State Data:
+++++++++++++++++++++++++++

#Ports #Ports #Ports #Ports #Ports #Ports #Ports
Admin-On Admin-Off Oper-On Oper-Off Off-Denied Off-No-PD Off-Fault
-------------------------------------------------------------------------
48 0 2 46 0 46 2



Cumulative Port Power Data:
+++++++++++++++++++++++++++

#Ports #Ports #Ports Power Power
Pri: 1 Pri: 2 Pri: 3 Consumption Allocation
-----------------------------------------------
0 0 48 8.400 W 44.932 W

show version

Copyright (c) Ruckus Networks, Inc. All rights reserved.
UNIT 1: compiled on Feb 8 2022 at 22:14:30 labeled as SPR09010a
(33554432 bytes) from Primary SPR09010a.bin (UFI)
SW: Version 09.0.10aT213
Compressed Primary Boot Code size = 786944, Version:10.1.22T215 (spz10122)
Compiled on Mon Jan 24 06:37:29 2022

HW: Stackable ICX7250-48-HPOE
==========================================================================
UNIT 1: SL 1: ICX7250-48P POE 48-port Management Module
Serial #:DUK384XXXXX
Software Package: ICX7250_L3_SOFT_PACKAGE
Current License: l3-prem-8X10G
P-ASIC 0: type B344, rev 01 Chip BCM56344_A0
==========================================================================
UNIT 1: SL 2: ICX7250-SFP-Plus 8-port 80G Module
==========================================================================
1000 MHz ARM processor ARMv7 88 MHz bus
8 MB boot flash memory
2 GB code flash memory
2 GB DRAM
STACKID 1 system uptime is 44 minute(s) 11 second(s)
The system started at 19:41:03 GMT-05 Mon Feb 28 2022

The system : started=warm start reloaded=by "reload"

show chassis

The stack unit 1 chassis info:

Power supply 1 (AC - PoE) present, status ok
Power supply 2 not present
Power supply 3 not present

Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 ok, speed (auto): [[1]]<->2
Fan 3 ok, speed (auto): [[1]]<->2

Fan controlled temperature:
Rule 1/2 (MGMT THERMAL PLANE): 66.3 deg-C
Rule 2/2 (AIR OUTLET NEAR PSU): 23.5 deg-C

Fan speed switching temperature thresholds:
Rule 1/2 (MGMT THERMAL PLANE):
Speed 1: NM<-----> 95 deg-C
Speed 2: 85<----->105 deg-C (shutdown)
Rule 2/2 (AIR OUTLET NEAR PSU):
Speed 1: NM<-----> 41 deg-C
Speed 2: 34<----->105 deg-C (shutdown)

Fan 1 Air Flow Direction: Front to Back
Fan 2 Air Flow Direction: Front to Back
Fan 3 Air Flow Direction: Front to Back
Slot 1 Current Temperature: 66.3 deg-C (Sensor 1), 24.5 deg-C (Sensor 2)
Slot 2 Current Temperature: NA
Warning level.......: 100.0 deg-C
Shutdown level......: 105.0 deg-C
 
Last edited:

fohdeesha

Kaini Industries
Nov 20, 2016
2,788
3,167
113
33
fohdeesha.com
I just picked up a secondhand 7250-48p with the help of the information in this thread. Thank you.

I am having an issue that POE does not work on some ports and I'm trying to figure out if there is a hardware issue or a software issue.

I followed the guide to get the switch updated and everything looked as I expected. Then I updated to the latest version 9.0.10a.

When I run show inline power detail debug-info I see (6) modules with HW version V1R3. However one module shows Vm2vErr as status and five show VOP-Sev1 as status. Only one is in Good status.

EDIT: I confirmed that the first 8 ports that are in Vm2vErr are the ports that are not powering POE devices.

Does anyone have any reference to these status codes and/or knows how to troubleshoot this? Google search turns up almost nothing.

FastIron Command Reference Guide has this information but I am not sure what causes these statuses or how to correct them.

The status of the device:
• Failed: No (bad or unreachable) PoE device found (bootup time)
• Good: OK - Expected PoE device Found (Zone1, no VOP)
• Revived: Device is currently refreshed
• n/a: Reserved
• Lost: Device lost or different from expected (while in operation)
• VOP-Sev1: Device error 1 (Zone2, VOP severity 1)
• VOP-Sev2: Device error 1 (Zone2, VOP severity 2)
• VOP-Sev3: Device error 1 (Zone2, VOP severity 3)
• VmErr: Device Vmain Error
• Vm2vErr: Device Vmain < System AVG Vmain by 2v or more
• Recovered: OK - Expected PoE device Found (Zone3, recovered from VOP)


Full output of show inline power detail debug-info:
Power Supply Data On unit 1:
++++++++++++++++++



Power Supply Data:
++++++++++++++++++

Power Supply #1:
Max Curr: 13.7 Amps
Voltage: 54.0 Volts
Capacity: 740 Watts
PoePower: 740 Watts
power supply 2 is not present
power supply 3 is not present


POE Details Info. On Unit 1 :


General PoE Data:
+++++++++++++++++

Firmware
Version
----------------
02.1.8 Build 004

Hardware
Version
----------------
V1R3

Device HW version : 0:V1R3 1:V1R3 2:V1R3 3:V1R3 4:V1R3 5:V1R3
Device Temperature(deg-C) : 0:42 1:40 2:40 3:42 4:37 5:39
Device Status : 0:Vm2vErr 1:VOP-Sev1 2:VOP-Sev1 3:VOP-Sev1 4:Good 5:VOP-Sev1



Cumulative Port State Data:
+++++++++++++++++++++++++++

#Ports #Ports #Ports #Ports #Ports #Ports #Ports
Admin-On Admin-Off Oper-On Oper-Off Off-Denied Off-No-PD Off-Fault
-------------------------------------------------------------------------
48 0 2 46 0 46 2



Cumulative Port Power Data:
+++++++++++++++++++++++++++

#Ports #Ports #Ports Power Power
Pri: 1 Pri: 2 Pri: 3 Consumption Allocation
-----------------------------------------------
0 0 48 8.400 W 44.932 W

show version

Copyright (c) Ruckus Networks, Inc. All rights reserved.
UNIT 1: compiled on Feb 8 2022 at 22:14:30 labeled as SPR09010a
(33554432 bytes) from Primary SPR09010a.bin (UFI)
SW: Version 09.0.10aT213
Compressed Primary Boot Code size = 786944, Version:10.1.22T215 (spz10122)
Compiled on Mon Jan 24 06:37:29 2022

HW: Stackable ICX7250-48-HPOE
==========================================================================
UNIT 1: SL 1: ICX7250-48P POE 48-port Management Module
Serial #:DUK384XXXXX
Software Package: ICX7250_L3_SOFT_PACKAGE
Current License: l3-prem-8X10G
P-ASIC 0: type B344, rev 01 Chip BCM56344_A0
==========================================================================
UNIT 1: SL 2: ICX7250-SFP-Plus 8-port 80G Module
==========================================================================
1000 MHz ARM processor ARMv7 88 MHz bus
8 MB boot flash memory
2 GB code flash memory
2 GB DRAM
STACKID 1 system uptime is 44 minute(s) 11 second(s)
The system started at 19:41:03 GMT-05 Mon Feb 28 2022

The system : started=warm start reloaded=by "reload"

show chassis

The stack unit 1 chassis info:

Power supply 1 (AC - PoE) present, status ok
Power supply 2 not present
Power supply 3 not present

Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 ok, speed (auto): [[1]]<->2
Fan 3 ok, speed (auto): [[1]]<->2

Fan controlled temperature:
Rule 1/2 (MGMT THERMAL PLANE): 66.3 deg-C
Rule 2/2 (AIR OUTLET NEAR PSU): 23.5 deg-C

Fan speed switching temperature thresholds:
Rule 1/2 (MGMT THERMAL PLANE):
Speed 1: NM<-----> 95 deg-C
Speed 2: 85<----->105 deg-C (shutdown)
Rule 2/2 (AIR OUTLET NEAR PSU):
Speed 1: NM<-----> 41 deg-C
Speed 2: 34<----->105 deg-C (shutdown)

Fan 1 Air Flow Direction: Front to Back
Fan 2 Air Flow Direction: Front to Back
Fan 3 Air Flow Direction: Front to Back
Slot 1 Current Temperature: 66.3 deg-C (Sensor 1), 24.5 deg-C (Sensor 2)
Slot 2 Current Temperature: NA
Warning level.......: 100.0 deg-C
Shutdown level......: 105.0 deg-C
hardware failure, get a return/replacement if you can
 
  • Like
Reactions: dwalme

fohdeesha

Kaini Industries
Nov 20, 2016
2,788
3,167
113
33
fohdeesha.com
Update: these Connectx-4 LX are running @1Gbps when connected to each other, and I can't get them to connect even directly with anything higher than that. My advice to the thread: stay very far away from Mellanox if you value your time.
I've used tons of different NIC vendors for a decade+ and can confidently say Mellanox is far and away the nicest to work with and the most reliable, I'd also wager it's what 90%+ users in this thread are using. I'd wager you either got bad cards, or more likely are on old or OEM-specific firmware and need to be reflashed
 

prdtabim

Active Member
Jan 29, 2022
179
69
28
Update: these Connectx-4 LX are running @1Gbps when connected to each other, and I can't get them to connect even directly with anything higher than that. My advice to the thread: stay very far away from Mellanox if you value your time.
Please show us "mlxconfig q" from both cards . Make sure that PHY_TYPE=XFI(2) and LINK_TYPE=ETH(2)
 

sko

Active Member
Jun 11, 2021
280
148
43
In case someone (like me) is searching for a comparison of the ICX 7250-48p noise level to cisco 3750X:

The 7250-48P is _a lot_ louder than a stack of 2 3750X (-48p & -12s, both /w NM-10G module) with 4 fans and 4 750W PSUs! The noise of the brocade is also much higher pitched, and therefore more present and annoying than the rather muffled rushing of the ciscos.
It also runs WAY hotter than the ciscos even if it was just idling vs the cisco stack under normal office loads.
I first compared it to my single 3750X-24P at home and was quite shocked that this is considered "almost silent" in brocade-terms. I then took it to the office where it now sat besides the (open) network cabinet with the cisco stack inside and it is definitely too loud to go anywhere someone can hear it.

So for (home)-office or homelab-use if you are in the same situation: just stay with the 3750X, even if you have to stack several of them they will still be quieter.

I wanted to upgrade from the 3750x-24p /w C3KX-NM-10G to the 7250-48P because it looked very promising and intriguing on paper, but I will now just add another 3750x with 10G module, so I will at least have 4x10G, and then I'll just wait another year or so until the 3850s have dropped in price a little more (those are even a bit quieter than the 3750x!).
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,788
3,167
113
33
fohdeesha.com
In case someone (like me) is searching for a comparison of the ICX 7250-48p noise level to cisco 3750X:

The 7250-48P is _a lot_ louder than a stack of 2 3750X (-48p & -12s, both /w NM-10G module) with 4 fans and 4 750W PSUs! The noise of the brocade is also much higher pitched, and therefore more present and annoying than the rather muffled rushing of the ciscos.
It also runs WAY hotter than the ciscos even if it was just idling vs the cisco stack under normal office loads.
I first compared it to my single 3750X-24P at home and was quite shocked that this is considered "almost silent" in brocade-terms. I then took it to the office where it now sat besides the (open) network cabinet with the cisco stack inside and it is definitely too loud to go anywhere someone can hear it.

So for (home)-office or homelab-use if you are in the same situation: just stay with the 3750X, even if you have to stack several of them they will still be quieter.

I wanted to upgrade from the 3750x-24p /w C3KX-NM-10G to the 7250-48P because it looked very promising and intriguing on paper, but I will now just add another 3750x with 10G module, so I will at least have 4x10G, and then I'll just wait another year or so until the 3850s have dropped in price a little more (those are even a bit quieter than the 3750x!).
the 48-port PoE version of the 7250 has 3 fans, versus 1 for the regular version. As covered previously in the thread, you can unplug all but 1 fan with no ill effects assuming you're not running tens of high powered PoE devices. Of course it runs hotter than your cisco, it has twice the ports (twice the ASIC), and twice the PoE per chassis volume. lastly, if it's that loud, the fan(s) in it might be on their way out with failing bearings, especially if it's "high pitched" like you say. I have an ICX7250-24 on the desk behind me and I literally cannot hear it over the laptop on top of it
 

bitbckt

will google compiler errors for scotch
Feb 22, 2022
216
135
43
I sit next to two ICX7250-24 in a stack all day with no problem.

IF you need to replace the fans, Gryphon GDA4028-12BB are great 40mm options.
 
  • Like
Reactions: dwalme

sko

Active Member
Jun 11, 2021
280
148
43
the 48-port PoE version of the 7250 has 3 fans, versus 1 for the regular version. As covered previously in the thread, you can unplug all but 1 fan with no ill effects assuming you're not running tens of high powered PoE devices. Of course it runs hotter than your cisco, it has twice the ports (twice the ASIC), and twice the PoE per chassis volume lastly, if it's that loud, the fan(s) in it might be on their way out with failing bearings, especially if it's "high pitched" like you say. I have an ICX7250-24 on the desk behind me and I literally cannot hear it over the laptop on top of it
Thanks for the input. I already tried unplugging one or more fans, which doesn't do much about the noise, but then the temperature rises within the first ~20 minutes above the threshold for fan speed 2 - i.e.: it gets even louder.

The comparisons (noise and temperatures) I mentioned were against the stack at the office, not the single -24p at home.
I am aware this switch COULD push a lot more packets than the 3750X, but at idle IMHO it should still be able to be as quiet as a stack with 48port PoE + 12 port SFP with 4 10GBit ports utilized under load.

The fans sound healthy and not like their bearings are dying, but I still might try to get some replacement fans. I've already searched for some options, but neither one of the Gryphoon or Deltas mentioned here are available here in germany/eu, let alone for a reasonable price from overseas (~40EUR+ / piece and even more for shipping sadly isn't "reasonable" - this would be over 50% of what the switch had cost...). I also searched for comparable models from Sunon or SanyoDenki, but except for OEM-volume-orders, especially the Sanys are next to impossible to get, or insanely expensive in single-digit quantities.
I already tried some Deltas, Sanyo Denki or Sunon I had lying around from servers or switches, but either the switch gets way too hot (and spins up the fans to even more annoying levels) or it won't be (much) quieter.
One major factor besides the fans seems to be the rather thin sheet metal that is used for the case - the top seems to resonate and amplify the fan noise, which gets better if some weight is applied. Also there is absolutely no guidance of the airflow, so most of the air seems to go from the PSU side straight to the fans, instead of being pulled over the ASICs and PoE board at the front of the switch. Don't know if this is normal or if there is some shroud missing inside of mine...


To be clear: given the raw performance data this thing still is dirt-cheap and "reasonably quiet" for something that should usually go into a server-room in the basement or in a datacenter.
But I think if noise is a point and/or you don't need this amount of throughput and *especially* if you already have one or more 3750X running, it isn't worth switching to the 7250 unless e.g. you absolutely need that amount of 10G ports (or the 40G ports on the 6610).
 

dwalme

New Member
Jan 16, 2018
5
0
1
45
I too was a little surprised at how loud the 7250-48p switch is that I just received. I expected it to be quieter based on the spec sheet and the information posted in this thread.

I could not sit with the switch in the same room as me. It's fine in my basement though.

In my case it'll be going back to the seller as my switch has a hardware failure with POE (unless the seller lets me keep it for cheap).
 

RoachedCoach

Member
Feb 4, 2020
39
46
18
Thanks for the input. I already tried unplugging one or more fans, which doesn't do much about the noise, but then the temperature rises within the first ~20 minutes above the threshold for fan speed 2 - i.e.: it gets even louder.

The comparisons (noise and temperatures) I mentioned were against the stack at the office, not the single -24p at home.
I am aware this switch COULD push a lot more packets than the 3750X, but at idle IMHO it should still be able to be as quiet as a stack with 48port PoE + 12 port SFP with 4 10GBit ports utilized under load.

The fans sound healthy and not like their bearings are dying, but I still might try to get some replacement fans. I've already searched for some options, but neither one of the Gryphoon or Deltas mentioned here are available here in germany/eu, let alone for a reasonable price from overseas (~40EUR+ / piece and even more for shipping sadly isn't "reasonable" - this would be over 50% of what the switch had cost...). I also searched for comparable models from Sunon or SanyoDenki, but except for OEM-volume-orders, especially the Sanys are next to impossible to get, or insanely expensive in single-digit quantities.
I already tried some Deltas, Sanyo Denki or Sunon I had lying around from servers or switches, but either the switch gets way too hot (and spins up the fans to even more annoying levels) or it won't be (much) quieter.
One major factor besides the fans seems to be the rather thin sheet metal that is used for the case - the top seems to resonate and amplify the fan noise, which gets better if some weight is applied. Also there is absolutely no guidance of the airflow, so most of the air seems to go from the PSU side straight to the fans, instead of being pulled over the ASICs and PoE board at the front of the switch. Don't know if this is normal or if there is some shroud missing inside of mine...


To be clear: given the raw performance data this thing still is dirt-cheap and "reasonably quiet" for something that should usually go into a server-room in the basement or in a datacenter.
But I think if noise is a point and/or you don't need this amount of throughput and *especially* if you already have one or more 3750X running, it isn't worth switching to the 7250 unless e.g. you absolutely need that amount of 10G ports (or the 40G ports on the 6610).
Wanted to plug my post on my fan changes to a 7250-48p.

 
  • Like
Reactions: dwalme

crackelf

Member
Apr 11, 2021
74
6
8
Please show us "mlxconfig q" from both cards . Make sure that PHY_TYPE=XFI(2) and LINK_TYPE=ETH(2)
No link type on this card. Already got the VF's working and loaded in Debian, but still nothing. I suspect the fiber or transceiver don't work with this card, despite advice.
Here's the card's current config:

Code:
Device type:    ConnectX4LX  
Name:           MCX4131A-GCA_Ax
Description:    ConnectX-4 Lx EN network interface card; 50GbE single-port QSFP28; PCIe3.0 x8; ROHS R6
Device:         03:00.0      

Configurations:                              Next Boot
         MEMIC_BAR_SIZE                      0            
         MEMIC_SIZE_LIMIT                    _256KB(1)    
         FLEX_PARSER_PROFILE_ENABLE          0            
         FLEX_IPV4_OVER_VXLAN_PORT           0            
         ROCE_NEXT_PROTOCOL                  254          
         NON_PREFETCHABLE_PF_BAR             False(0)     
         VF_VPD_ENABLE                       False(0)     
         STRICT_VF_MSIX_NUM                  False(0)     
         VF_NODNIC_ENABLE                    False(0)     
         NUM_OF_VFS                          8            
         SRIOV_EN                            True(1)      
         PF_LOG_BAR_SIZE                     5            
         VF_LOG_BAR_SIZE                     0            
         NUM_PF_MSIX                         63           
         NUM_VF_MSIX                         11           
         INT_LOG_MAX_PAYLOAD_SIZE            AUTOMATIC(0) 
         PCIE_CREDIT_TOKEN_TIMEOUT           0            
         ACCURATE_TX_SCHEDULER               False(0)     
         PARTIAL_RESET_EN                    False(0)     
         SW_RECOVERY_ON_ERRORS               False(0)     
         RESET_WITH_HOST_ON_ERRORS           False(0)     
         PCI_BUS0_RESTRICT_SPEED             PCI_GEN_1(0) 
         PCI_BUS0_RESTRICT_ASPM              False(0)     
         PCI_BUS0_RESTRICT_WIDTH             PCI_X1(0)    
         PCI_BUS0_RESTRICT                   False(0)     
         CQE_COMPRESSION                     BALANCED(0)  
         IP_OVER_VXLAN_EN                    False(0)     
         MKEY_BY_NAME                        False(0)     
         UCTX_EN                             True(1)      
         PCI_ATOMIC_MODE                     PCI_ATOMIC_DISABLED_EXT_ATOMIC_ENABLED(0)
         TUNNEL_ECN_COPY_DISABLE             False(0)     
         LRO_LOG_TIMEOUT0                    6            
         LRO_LOG_TIMEOUT1                    7            
         LRO_LOG_TIMEOUT2                    8            
         LRO_LOG_TIMEOUT3                    13           
         TX_SCHEDULER_BURST                  0            
         LOG_DCR_HASH_TABLE_SIZE             14           
         DCR_LIFO_SIZE                       16384        
         ROCE_CC_PRIO_MASK_P1                255          
         CLAMP_TGT_RATE_AFTER_TIME_INC_P1    True(1)      
         CLAMP_TGT_RATE_P1                   False(0)     
         RPG_TIME_RESET_P1                   300          
         RPG_BYTE_RESET_P1                   32767        
         RPG_THRESHOLD_P1                    1            
         RPG_MAX_RATE_P1                     0            
         RPG_AI_RATE_P1                      5            
         RPG_HAI_RATE_P1                     50           
         RPG_GD_P1                           11           
         RPG_MIN_DEC_FAC_P1                  50           
         RPG_MIN_RATE_P1                     1            
         RATE_TO_SET_ON_FIRST_CNP_P1         0            
         DCE_TCP_G_P1                        1019         
         DCE_TCP_RTT_P1                      1            
         RATE_REDUCE_MONITOR_PERIOD_P1       4            
         INITIAL_ALPHA_VALUE_P1              1023         
         MIN_TIME_BETWEEN_CNPS_P1            4            
         CNP_802P_PRIO_P1                    6            
         CNP_DSCP_P1                         48           
         LLDP_NB_DCBX_P1                     False(0)     
         LLDP_NB_RX_MODE_P1                  OFF(0)       
         LLDP_NB_TX_MODE_P1                  OFF(0)       
         DCBX_IEEE_P1                        True(1)      
         DCBX_CEE_P1                         True(1)      
         DCBX_WILLING_P1                     True(1)      
         KEEP_ETH_LINK_UP_P1                 True(1)      
         KEEP_IB_LINK_UP_P1                  False(0)     
         KEEP_LINK_UP_ON_BOOT_P1             False(0)     
         KEEP_LINK_UP_ON_STANDBY_P1          False(0)     
         DO_NOT_CLEAR_PORT_STATS_P1          False(0)     
         AUTO_POWER_SAVE_LINK_DOWN_P1        False(0)     
         NUM_OF_VL_P1                        _4_VLs(3)    
         NUM_OF_TC_P1                        _8_TCs(0)    
         NUM_OF_PFC_P1                       8            
         VL15_BUFFER_SIZE_P1                 0            
         DUP_MAC_ACTION_P1                   LAST_CFG(0)  
         SRIOV_IB_ROUTING_MODE_P1            LID(1)       
         IB_ROUTING_MODE_P1                  LID(1)       
         PCI_WR_ORDERING                     per_mkey(0)  
         MULTI_PORT_VHCA_EN                  False(0)     
         PORT_OWNER                          True(1)      
         ALLOW_RD_COUNTERS                   True(1)      
         RENEG_ON_CHANGE                     True(1)      
         TRACER_ENABLE                       True(1)      
         IP_VER                              IPv4(0)      
         UEFI_HII_EN                         True(1)      
         BOOT_DBG_LOG                        False(0)     
         UEFI_LOGS                           DISABLED(0)  
         BOOT_INTERRUPT_DIS                  False(0)     
         BOOT_LACP_DIS                       True(1)      
         DYNAMIC_VF_MSIX_TABLE               False(0)     
         EXP_ROM_UEFI_ARM_ENABLE             False(0)     
         EXP_ROM_UEFI_x86_ENABLE             False(0)     
         EXP_ROM_PXE_ENABLE                  True(1)      
         ADVANCED_PCI_SETTINGS               False(0)     
         SAFE_MODE_THRESHOLD                 10           
         SAFE_MODE_ENABLE                    True(1)
Just to clarify my setup here in case anything looks wrong:
OS2 LC-LC Singlemode Duplex
KAIAM QSFP+40G-LR4 lite
Mellanox MCX4131A-GCAT_C05