Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

crackelf

Member
Apr 11, 2021
74
6
8
hmm maybe try removing the "stack disable" option, (either "no stack disable" or "stack enable" can't remember which) then rebooting. then if you can paste the output of (assuming 1/3/1 is actually plugged into a card):

show int e 1/3/1
show media ethernet 1/3/1
show optic 1/3/1
Update: these Connectx-4 LX are running @1Gbps when connected to each other, and I can't get them to connect even directly with anything higher than that. My advice to the thread: stay very far away from Mellanox if you value your time.
 
Last edited:

simbo

New Member
Feb 24, 2022
10
2
3
Got a small problem where I can't get the 10GB breakout interfaces to come up. I'm trying to brake out the 40GB to 10GB LCs using some FS.com gear. I've followed the setup guide for removing the ports from the stack.

To get the 10G LCs on the front of the rack, I've purchased the following:
- QSFP-SR4-40G Brocade 40G-QSFP-SR4 Compatible 40GBASE-SR4 QSFP+ 850nm 150m DOM MTP/MPO MMF Optical Transceiver Module
- 12FMTPOM4 1m (3ft) MTP®-12 (Female) to MTP®-12 (Female) OM4 Multimode Elite Trunk Cable, 12 Fibers, Type B, Plenum (OFNP), Magenta
-
FHD-1MTP4LCDOM4 FHD MTP®-8 Cassette, 8 Fibers OM4 Multimode, Universal Polarity, MTP® to 4 x LC Duplex (Aqua), 0.35dB max

I'm using FS.com 10GB 850nm SR transceiver's on the other end with out any problems over OM4 cables.


Code:
SSH@sw-core(config)#show media validation


Port       Supported Vendor               Type
----------------------------------------------------------------------
1/2/2      Yes       BROCADE              40G QSFP-SR4
1/2/3      Yes       BROCADE              40G QSFP-SR4
1/2/4      Yes       BROCADE              40G QSFP-SR4
1/2/5      Yes       BROCADE              40G QSFP-SR4
1/2/7      Yes       BROCADE              40G QSFP-SR4
1/2/8      Yes       BROCADE              40G QSFP-SR4
1/2/9      Yes       BROCADE              40G QSFP-SR4
1/2/10     Yes       BROCADE              40G QSFP-SR4
1/3/3      Yes       FS                    Type  : 10GE  Passive Twinax  1m (SFP +) (Not supported)
1/3/4      Yes       FS                    Type  : 10GE  Passive Twinax  1m (SFP +) (Not supported)
1/3/5      Yes       FS                    Type  : 10GE  Passive Twinax  5m (SFP +) (Not supported)
1/3/6      Yes       FS                    Type  : 10GE  Passive Twinax  5m (SFP +) (Not supported)
1/3/7      Yes       OEM                   Type  : 10GE SR 300m (SFP +)

1/3/7 is an optic chucked into the front port used to test the LC cables. The other 1/3's are LAGs go to other switchgear.


Code:
SSH@sw-core(config)#show conf
!
Startup-config data location is flash memory
!
Startup configuration:
!
ver 08.0.30uT7f3
!
stack unit 1
  module 1 icx6610-48p-poe-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
stack disable

FWIW, I set the 1/2/2 to 1/2/5 and 1/2/7 to 1/2/10 to speed-duplex 10G-full also left blank.

Here's the interface summary

Code:
1/2/1      Down    None    None None  None  Yes 1    0   748e.f8fe.c148
1/2/2      Down    None    None None  None  No  1    0   748e.f8fe.c148
1/2/3      Down    None    None None  None  Yes 1    0   748e.f8fe.c148
1/2/4      Down    None    None None  None  No  1    0   748e.f8fe.c148
1/2/5      Down    None    None None  None  No  1    0   748e.f8fe.c148
1/2/6      Down    None    None None  None  No  1    0   748e.f8fe.c148
1/2/7      Down    None    None None  None  Yes 1    0   748e.f8fe.c148  SW-XXX1
1/2/8      Down    None    None None  None  Yes 1    0   748e.f8fe.c148  SW-XXX2
1/2/9      Down    None    None None  None  Yes 1    0   748e.f8fe.c148  SW-XXX3
1/2/10     Down    None    None None  None  Yes 1    0   748e.f8fe.c148
1/3/1      Down    None    None None  None  Yes 1    0   748e.f8fe.c148
1/3/2      Down    None    None None  None  Yes 1    0   748e.f8fe.c148
1/3/3      Up      Forward Full 10G   3     Yes 1    0   748e.f8fe.c148  SW-DXX-1
1/3/4      Up      Forward Full 10G   3     Yes 1    0   748e.f8fe.c148  SW-DXX-2
1/3/5      Up      Forward Full 10G   2     Yes 1    0   748e.f8fe.c148  SW-RXX-1
1/3/6      Up      Forward Full 10G   2     Yes 1    0   748e.f8fe.c148  SW-RXX-2
1/3/7      Down    None    None None  None  Yes 1    0   748e.f8fe.c148
1/3/8      Down    None    None None  None  Yes 1    0   748e.f8fe.c148
Here's the interface that has a plugged in LC cable:
Code:
SSH@sw-core(config)#show int ethe 1/2/7
  10GigabitEthernet 1/2/7 is down, line protocol is down
  Port down for 2 hour(s) 25 minute(s) 13 second(s)
  Hardware is   10GigabitEthernet , address is 748e.f8fe.c148 (bia 748e.f8fe.c17f)
  Configured speed 10Gbit, actual unknown, configured duplex fdx, actual unknown
  Configured mdi mode AUTO, actual unknown
  Member of 11 L2 VLANs, port is dual mode in Vlan 1, port state is BLOCKING
  BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
  Link Error Dampening is Disabled
  STP configured to ON, priority is level0, mac-learning is enabled
  Openflow is Disabled, Openflow Hybrid mode is Disabled,  Flow Control is config enabled, oper enabled, negotiation disabled
  Mirror disabled, Monitor disabled
  Mac-notification is disabled
  Not member of any active trunks
  Not member of any configured trunks
  Port name is SW-XXX1
  MTU 10200 bytes, encapsulation ethernet
  300 second input rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
  300 second output rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
  54035 packets input, 15271970 bytes, 0 no buffer
  Received 2649 broadcasts, 3205 multicasts, 48181 unicasts
  0 input errors, 0 CRC, 0 frame, 0 ignored
  0 runts, 0 giants
  129068 packets output, 41755954 bytes, 0 underruns
  Transmitted 11005 broadcasts, 55975 multicasts, 62088 unicasts
  0 output errors, 0 collisions
  Relay Agent Information option: Disabled

Egress queues:
Queue counters    Queued packets    Dropped Packets
    0              115303                   0
    1                   0                   0
    2                   0                   0
    3                   0                   0
    4                   0                   0
    5                   1                   0
    6                   0                   0
Code:
SSH@sw-core>show ver
  Copyright (c) 1996-2016 Brocade Communications Systems, Inc. All rights reserved.
    UNIT 1: compiled on Apr 23 2020 at 13:17:12 labeled as FCXR08030u
                (10545591 bytes) from Primary FCXR08030u.bin
        SW: Version 08.0.30uT7f3
  Boot-Monitor Image size = 370695, Version:10.1.00T7f5 (grz10100)
  HW: Stackable ICX6610-48-HPOE
==========================================================================
UNIT 1: SL 1: ICX6610-48P POE 48-port Management Module
         Serial  #: 2ax5o2jk68e
         License: ICX6610_ADV_ROUTER_SOFT_PACKAGE   (LID: H4CKTH3PLN8)
         P-ENGINE  0: type E02B, rev 01
         P-ENGINE  1: type E02B, rev 01
==========================================================================
UNIT 1: SL 2: ICX6610-QSFP 10-port 160G Module
==========================================================================
UNIT 1: SL 3: ICX6610-8-port Dual Mode(SFP/SFP+) Module
==========================================================================
  800 MHz Power PC processor 8544E (version 0021/0023) 400 MHz bus
65536 KB flash memory
  512 MB DRAM
STACKID 1  system uptime is 5 day(s) 1 hour(s) 38 minute(s) 27 second(s)
The system started at 12:06:39 GMT+10 Sun Feb 20 2022

The system : started=cold start
I've double checked that the cables are plugged in. I have 2 x of the above kit and I get the same behaviour on each 40GQSFP, MTP cable and breakout cassette.

Any ideas on what to try next?
OK.....I reset the speed-duplex to auto and did a cold boot and the interfaces came up as 10G; i.e. working.
 

dwalme

New Member
Jan 16, 2018
6
0
1
45
I just picked up a secondhand 7250-48p with the help of the information in this thread. Thank you.

I am having an issue that POE does not work on some ports and I'm trying to figure out if there is a hardware issue or a software issue.

I followed the guide to get the switch updated and everything looked as I expected. Then I updated to the latest version 9.0.10a.

When I run show inline power detail debug-info I see (6) modules with HW version V1R3. However one module shows Vm2vErr as status and five show VOP-Sev1 as status. Only one is in Good status.

EDIT: I confirmed that the first 8 ports that are in Vm2vErr are the ports that are not powering POE devices.

Does anyone have any reference to these status codes and/or knows how to troubleshoot this? Google search turns up almost nothing.

FastIron Command Reference Guide has this information but I am not sure what causes these statuses or how to correct them.

The status of the device:
• Failed: No (bad or unreachable) PoE device found (bootup time)
• Good: OK - Expected PoE device Found (Zone1, no VOP)
• Revived: Device is currently refreshed
• n/a: Reserved
• Lost: Device lost or different from expected (while in operation)
• VOP-Sev1: Device error 1 (Zone2, VOP severity 1)
• VOP-Sev2: Device error 1 (Zone2, VOP severity 2)
• VOP-Sev3: Device error 1 (Zone2, VOP severity 3)
• VmErr: Device Vmain Error
• Vm2vErr: Device Vmain < System AVG Vmain by 2v or more
• Recovered: OK - Expected PoE device Found (Zone3, recovered from VOP)


Full output of show inline power detail debug-info:
Power Supply Data On unit 1:
++++++++++++++++++



Power Supply Data:
++++++++++++++++++

Power Supply #1:
Max Curr: 13.7 Amps
Voltage: 54.0 Volts
Capacity: 740 Watts
PoePower: 740 Watts
power supply 2 is not present
power supply 3 is not present


POE Details Info. On Unit 1 :


General PoE Data:
+++++++++++++++++

Firmware
Version
----------------
02.1.8 Build 004

Hardware
Version
----------------
V1R3

Device HW version : 0:V1R3 1:V1R3 2:V1R3 3:V1R3 4:V1R3 5:V1R3
Device Temperature(deg-C) : 0:42 1:40 2:40 3:42 4:37 5:39
Device Status : 0:Vm2vErr 1:VOP-Sev1 2:VOP-Sev1 3:VOP-Sev1 4:Good 5:VOP-Sev1



Cumulative Port State Data:
+++++++++++++++++++++++++++

#Ports #Ports #Ports #Ports #Ports #Ports #Ports
Admin-On Admin-Off Oper-On Oper-Off Off-Denied Off-No-PD Off-Fault
-------------------------------------------------------------------------
48 0 2 46 0 46 2



Cumulative Port Power Data:
+++++++++++++++++++++++++++

#Ports #Ports #Ports Power Power
Pri: 1 Pri: 2 Pri: 3 Consumption Allocation
-----------------------------------------------
0 0 48 8.400 W 44.932 W

show version

Copyright (c) Ruckus Networks, Inc. All rights reserved.
UNIT 1: compiled on Feb 8 2022 at 22:14:30 labeled as SPR09010a
(33554432 bytes) from Primary SPR09010a.bin (UFI)
SW: Version 09.0.10aT213
Compressed Primary Boot Code size = 786944, Version:10.1.22T215 (spz10122)
Compiled on Mon Jan 24 06:37:29 2022

HW: Stackable ICX7250-48-HPOE
==========================================================================
UNIT 1: SL 1: ICX7250-48P POE 48-port Management Module
Serial #:DUK384XXXXX
Software Package: ICX7250_L3_SOFT_PACKAGE
Current License: l3-prem-8X10G
P-ASIC 0: type B344, rev 01 Chip BCM56344_A0
==========================================================================
UNIT 1: SL 2: ICX7250-SFP-Plus 8-port 80G Module
==========================================================================
1000 MHz ARM processor ARMv7 88 MHz bus
8 MB boot flash memory
2 GB code flash memory
2 GB DRAM
STACKID 1 system uptime is 44 minute(s) 11 second(s)
The system started at 19:41:03 GMT-05 Mon Feb 28 2022

The system : started=warm start reloaded=by "reload"

show chassis

The stack unit 1 chassis info:

Power supply 1 (AC - PoE) present, status ok
Power supply 2 not present
Power supply 3 not present

Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 ok, speed (auto): [[1]]<->2
Fan 3 ok, speed (auto): [[1]]<->2

Fan controlled temperature:
Rule 1/2 (MGMT THERMAL PLANE): 66.3 deg-C
Rule 2/2 (AIR OUTLET NEAR PSU): 23.5 deg-C

Fan speed switching temperature thresholds:
Rule 1/2 (MGMT THERMAL PLANE):
Speed 1: NM<-----> 95 deg-C
Speed 2: 85<----->105 deg-C (shutdown)
Rule 2/2 (AIR OUTLET NEAR PSU):
Speed 1: NM<-----> 41 deg-C
Speed 2: 34<----->105 deg-C (shutdown)

Fan 1 Air Flow Direction: Front to Back
Fan 2 Air Flow Direction: Front to Back
Fan 3 Air Flow Direction: Front to Back
Slot 1 Current Temperature: 66.3 deg-C (Sensor 1), 24.5 deg-C (Sensor 2)
Slot 2 Current Temperature: NA
Warning level.......: 100.0 deg-C
Shutdown level......: 105.0 deg-C
 
Last edited:

fohdeesha

Kaini Industries
Nov 20, 2016
2,869
3,372
113
34
fohdeesha.com
I just picked up a secondhand 7250-48p with the help of the information in this thread. Thank you.

I am having an issue that POE does not work on some ports and I'm trying to figure out if there is a hardware issue or a software issue.

I followed the guide to get the switch updated and everything looked as I expected. Then I updated to the latest version 9.0.10a.

When I run show inline power detail debug-info I see (6) modules with HW version V1R3. However one module shows Vm2vErr as status and five show VOP-Sev1 as status. Only one is in Good status.

EDIT: I confirmed that the first 8 ports that are in Vm2vErr are the ports that are not powering POE devices.

Does anyone have any reference to these status codes and/or knows how to troubleshoot this? Google search turns up almost nothing.

FastIron Command Reference Guide has this information but I am not sure what causes these statuses or how to correct them.

The status of the device:
• Failed: No (bad or unreachable) PoE device found (bootup time)
• Good: OK - Expected PoE device Found (Zone1, no VOP)
• Revived: Device is currently refreshed
• n/a: Reserved
• Lost: Device lost or different from expected (while in operation)
• VOP-Sev1: Device error 1 (Zone2, VOP severity 1)
• VOP-Sev2: Device error 1 (Zone2, VOP severity 2)
• VOP-Sev3: Device error 1 (Zone2, VOP severity 3)
• VmErr: Device Vmain Error
• Vm2vErr: Device Vmain < System AVG Vmain by 2v or more
• Recovered: OK - Expected PoE device Found (Zone3, recovered from VOP)


Full output of show inline power detail debug-info:
Power Supply Data On unit 1:
++++++++++++++++++



Power Supply Data:
++++++++++++++++++

Power Supply #1:
Max Curr: 13.7 Amps
Voltage: 54.0 Volts
Capacity: 740 Watts
PoePower: 740 Watts
power supply 2 is not present
power supply 3 is not present


POE Details Info. On Unit 1 :


General PoE Data:
+++++++++++++++++

Firmware
Version
----------------
02.1.8 Build 004

Hardware
Version
----------------
V1R3

Device HW version : 0:V1R3 1:V1R3 2:V1R3 3:V1R3 4:V1R3 5:V1R3
Device Temperature(deg-C) : 0:42 1:40 2:40 3:42 4:37 5:39
Device Status : 0:Vm2vErr 1:VOP-Sev1 2:VOP-Sev1 3:VOP-Sev1 4:Good 5:VOP-Sev1



Cumulative Port State Data:
+++++++++++++++++++++++++++

#Ports #Ports #Ports #Ports #Ports #Ports #Ports
Admin-On Admin-Off Oper-On Oper-Off Off-Denied Off-No-PD Off-Fault
-------------------------------------------------------------------------
48 0 2 46 0 46 2



Cumulative Port Power Data:
+++++++++++++++++++++++++++

#Ports #Ports #Ports Power Power
Pri: 1 Pri: 2 Pri: 3 Consumption Allocation
-----------------------------------------------
0 0 48 8.400 W 44.932 W

show version

Copyright (c) Ruckus Networks, Inc. All rights reserved.
UNIT 1: compiled on Feb 8 2022 at 22:14:30 labeled as SPR09010a
(33554432 bytes) from Primary SPR09010a.bin (UFI)
SW: Version 09.0.10aT213
Compressed Primary Boot Code size = 786944, Version:10.1.22T215 (spz10122)
Compiled on Mon Jan 24 06:37:29 2022

HW: Stackable ICX7250-48-HPOE
==========================================================================
UNIT 1: SL 1: ICX7250-48P POE 48-port Management Module
Serial #:DUK384XXXXX
Software Package: ICX7250_L3_SOFT_PACKAGE
Current License: l3-prem-8X10G
P-ASIC 0: type B344, rev 01 Chip BCM56344_A0
==========================================================================
UNIT 1: SL 2: ICX7250-SFP-Plus 8-port 80G Module
==========================================================================
1000 MHz ARM processor ARMv7 88 MHz bus
8 MB boot flash memory
2 GB code flash memory
2 GB DRAM
STACKID 1 system uptime is 44 minute(s) 11 second(s)
The system started at 19:41:03 GMT-05 Mon Feb 28 2022

The system : started=warm start reloaded=by "reload"

show chassis

The stack unit 1 chassis info:

Power supply 1 (AC - PoE) present, status ok
Power supply 2 not present
Power supply 3 not present

Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 ok, speed (auto): [[1]]<->2
Fan 3 ok, speed (auto): [[1]]<->2

Fan controlled temperature:
Rule 1/2 (MGMT THERMAL PLANE): 66.3 deg-C
Rule 2/2 (AIR OUTLET NEAR PSU): 23.5 deg-C

Fan speed switching temperature thresholds:
Rule 1/2 (MGMT THERMAL PLANE):
Speed 1: NM<-----> 95 deg-C
Speed 2: 85<----->105 deg-C (shutdown)
Rule 2/2 (AIR OUTLET NEAR PSU):
Speed 1: NM<-----> 41 deg-C
Speed 2: 34<----->105 deg-C (shutdown)

Fan 1 Air Flow Direction: Front to Back
Fan 2 Air Flow Direction: Front to Back
Fan 3 Air Flow Direction: Front to Back
Slot 1 Current Temperature: 66.3 deg-C (Sensor 1), 24.5 deg-C (Sensor 2)
Slot 2 Current Temperature: NA
Warning level.......: 100.0 deg-C
Shutdown level......: 105.0 deg-C
hardware failure, get a return/replacement if you can
 
  • Like
Reactions: dwalme

fohdeesha

Kaini Industries
Nov 20, 2016
2,869
3,372
113
34
fohdeesha.com
Update: these Connectx-4 LX are running @1Gbps when connected to each other, and I can't get them to connect even directly with anything higher than that. My advice to the thread: stay very far away from Mellanox if you value your time.
I've used tons of different NIC vendors for a decade+ and can confidently say Mellanox is far and away the nicest to work with and the most reliable, I'd also wager it's what 90%+ users in this thread are using. I'd wager you either got bad cards, or more likely are on old or OEM-specific firmware and need to be reflashed
 

prdtabim

Active Member
Jan 29, 2022
188
73
28
Update: these Connectx-4 LX are running @1Gbps when connected to each other, and I can't get them to connect even directly with anything higher than that. My advice to the thread: stay very far away from Mellanox if you value your time.
Please show us "mlxconfig q" from both cards . Make sure that PHY_TYPE=XFI(2) and LINK_TYPE=ETH(2)
 

sko

Active Member
Jun 11, 2021
367
223
43
In case someone (like me) is searching for a comparison of the ICX 7250-48p noise level to cisco 3750X:

The 7250-48P is _a lot_ louder than a stack of 2 3750X (-48p & -12s, both /w NM-10G module) with 4 fans and 4 750W PSUs! The noise of the brocade is also much higher pitched, and therefore more present and annoying than the rather muffled rushing of the ciscos.
It also runs WAY hotter than the ciscos even if it was just idling vs the cisco stack under normal office loads.
I first compared it to my single 3750X-24P at home and was quite shocked that this is considered "almost silent" in brocade-terms. I then took it to the office where it now sat besides the (open) network cabinet with the cisco stack inside and it is definitely too loud to go anywhere someone can hear it.

So for (home)-office or homelab-use if you are in the same situation: just stay with the 3750X, even if you have to stack several of them they will still be quieter.

I wanted to upgrade from the 3750x-24p /w C3KX-NM-10G to the 7250-48P because it looked very promising and intriguing on paper, but I will now just add another 3750x with 10G module, so I will at least have 4x10G, and then I'll just wait another year or so until the 3850s have dropped in price a little more (those are even a bit quieter than the 3750x!).
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,869
3,372
113
34
fohdeesha.com
In case someone (like me) is searching for a comparison of the ICX 7250-48p noise level to cisco 3750X:

The 7250-48P is _a lot_ louder than a stack of 2 3750X (-48p & -12s, both /w NM-10G module) with 4 fans and 4 750W PSUs! The noise of the brocade is also much higher pitched, and therefore more present and annoying than the rather muffled rushing of the ciscos.
It also runs WAY hotter than the ciscos even if it was just idling vs the cisco stack under normal office loads.
I first compared it to my single 3750X-24P at home and was quite shocked that this is considered "almost silent" in brocade-terms. I then took it to the office where it now sat besides the (open) network cabinet with the cisco stack inside and it is definitely too loud to go anywhere someone can hear it.

So for (home)-office or homelab-use if you are in the same situation: just stay with the 3750X, even if you have to stack several of them they will still be quieter.

I wanted to upgrade from the 3750x-24p /w C3KX-NM-10G to the 7250-48P because it looked very promising and intriguing on paper, but I will now just add another 3750x with 10G module, so I will at least have 4x10G, and then I'll just wait another year or so until the 3850s have dropped in price a little more (those are even a bit quieter than the 3750x!).
the 48-port PoE version of the 7250 has 3 fans, versus 1 for the regular version. As covered previously in the thread, you can unplug all but 1 fan with no ill effects assuming you're not running tens of high powered PoE devices. Of course it runs hotter than your cisco, it has twice the ports (twice the ASIC), and twice the PoE per chassis volume. lastly, if it's that loud, the fan(s) in it might be on their way out with failing bearings, especially if it's "high pitched" like you say. I have an ICX7250-24 on the desk behind me and I literally cannot hear it over the laptop on top of it
 

bitbckt

will google compiler errors for scotch
Feb 22, 2022
216
136
43
I sit next to two ICX7250-24 in a stack all day with no problem.

IF you need to replace the fans, Gryphon GDA4028-12BB are great 40mm options.
 
  • Like
Reactions: dwalme

sko

Active Member
Jun 11, 2021
367
223
43
the 48-port PoE version of the 7250 has 3 fans, versus 1 for the regular version. As covered previously in the thread, you can unplug all but 1 fan with no ill effects assuming you're not running tens of high powered PoE devices. Of course it runs hotter than your cisco, it has twice the ports (twice the ASIC), and twice the PoE per chassis volume lastly, if it's that loud, the fan(s) in it might be on their way out with failing bearings, especially if it's "high pitched" like you say. I have an ICX7250-24 on the desk behind me and I literally cannot hear it over the laptop on top of it
Thanks for the input. I already tried unplugging one or more fans, which doesn't do much about the noise, but then the temperature rises within the first ~20 minutes above the threshold for fan speed 2 - i.e.: it gets even louder.

The comparisons (noise and temperatures) I mentioned were against the stack at the office, not the single -24p at home.
I am aware this switch COULD push a lot more packets than the 3750X, but at idle IMHO it should still be able to be as quiet as a stack with 48port PoE + 12 port SFP with 4 10GBit ports utilized under load.

The fans sound healthy and not like their bearings are dying, but I still might try to get some replacement fans. I've already searched for some options, but neither one of the Gryphoon or Deltas mentioned here are available here in germany/eu, let alone for a reasonable price from overseas (~40EUR+ / piece and even more for shipping sadly isn't "reasonable" - this would be over 50% of what the switch had cost...). I also searched for comparable models from Sunon or SanyoDenki, but except for OEM-volume-orders, especially the Sanys are next to impossible to get, or insanely expensive in single-digit quantities.
I already tried some Deltas, Sanyo Denki or Sunon I had lying around from servers or switches, but either the switch gets way too hot (and spins up the fans to even more annoying levels) or it won't be (much) quieter.
One major factor besides the fans seems to be the rather thin sheet metal that is used for the case - the top seems to resonate and amplify the fan noise, which gets better if some weight is applied. Also there is absolutely no guidance of the airflow, so most of the air seems to go from the PSU side straight to the fans, instead of being pulled over the ASICs and PoE board at the front of the switch. Don't know if this is normal or if there is some shroud missing inside of mine...


To be clear: given the raw performance data this thing still is dirt-cheap and "reasonably quiet" for something that should usually go into a server-room in the basement or in a datacenter.
But I think if noise is a point and/or you don't need this amount of throughput and *especially* if you already have one or more 3750X running, it isn't worth switching to the 7250 unless e.g. you absolutely need that amount of 10G ports (or the 40G ports on the 6610).
 

dwalme

New Member
Jan 16, 2018
6
0
1
45
I too was a little surprised at how loud the 7250-48p switch is that I just received. I expected it to be quieter based on the spec sheet and the information posted in this thread.

I could not sit with the switch in the same room as me. It's fine in my basement though.

In my case it'll be going back to the seller as my switch has a hardware failure with POE (unless the seller lets me keep it for cheap).
 

RoachedCoach

Member
Feb 4, 2020
46
54
18
Thanks for the input. I already tried unplugging one or more fans, which doesn't do much about the noise, but then the temperature rises within the first ~20 minutes above the threshold for fan speed 2 - i.e.: it gets even louder.

The comparisons (noise and temperatures) I mentioned were against the stack at the office, not the single -24p at home.
I am aware this switch COULD push a lot more packets than the 3750X, but at idle IMHO it should still be able to be as quiet as a stack with 48port PoE + 12 port SFP with 4 10GBit ports utilized under load.

The fans sound healthy and not like their bearings are dying, but I still might try to get some replacement fans. I've already searched for some options, but neither one of the Gryphoon or Deltas mentioned here are available here in germany/eu, let alone for a reasonable price from overseas (~40EUR+ / piece and even more for shipping sadly isn't "reasonable" - this would be over 50% of what the switch had cost...). I also searched for comparable models from Sunon or SanyoDenki, but except for OEM-volume-orders, especially the Sanys are next to impossible to get, or insanely expensive in single-digit quantities.
I already tried some Deltas, Sanyo Denki or Sunon I had lying around from servers or switches, but either the switch gets way too hot (and spins up the fans to even more annoying levels) or it won't be (much) quieter.
One major factor besides the fans seems to be the rather thin sheet metal that is used for the case - the top seems to resonate and amplify the fan noise, which gets better if some weight is applied. Also there is absolutely no guidance of the airflow, so most of the air seems to go from the PSU side straight to the fans, instead of being pulled over the ASICs and PoE board at the front of the switch. Don't know if this is normal or if there is some shroud missing inside of mine...


To be clear: given the raw performance data this thing still is dirt-cheap and "reasonably quiet" for something that should usually go into a server-room in the basement or in a datacenter.
But I think if noise is a point and/or you don't need this amount of throughput and *especially* if you already have one or more 3750X running, it isn't worth switching to the 7250 unless e.g. you absolutely need that amount of 10G ports (or the 40G ports on the 6610).
Wanted to plug my post on my fan changes to a 7250-48p.

 
  • Like
Reactions: dwalme

crackelf

Member
Apr 11, 2021
74
6
8
Please show us "mlxconfig q" from both cards . Make sure that PHY_TYPE=XFI(2) and LINK_TYPE=ETH(2)
No link type on this card. Already got the VF's working and loaded in Debian, but still nothing. I suspect the fiber or transceiver don't work with this card, despite advice.
Here's the card's current config:

Code:
Device type:    ConnectX4LX  
Name:           MCX4131A-GCA_Ax
Description:    ConnectX-4 Lx EN network interface card; 50GbE single-port QSFP28; PCIe3.0 x8; ROHS R6
Device:         03:00.0      

Configurations:                              Next Boot
         MEMIC_BAR_SIZE                      0            
         MEMIC_SIZE_LIMIT                    _256KB(1)    
         FLEX_PARSER_PROFILE_ENABLE          0            
         FLEX_IPV4_OVER_VXLAN_PORT           0            
         ROCE_NEXT_PROTOCOL                  254          
         NON_PREFETCHABLE_PF_BAR             False(0)     
         VF_VPD_ENABLE                       False(0)     
         STRICT_VF_MSIX_NUM                  False(0)     
         VF_NODNIC_ENABLE                    False(0)     
         NUM_OF_VFS                          8            
         SRIOV_EN                            True(1)      
         PF_LOG_BAR_SIZE                     5            
         VF_LOG_BAR_SIZE                     0            
         NUM_PF_MSIX                         63           
         NUM_VF_MSIX                         11           
         INT_LOG_MAX_PAYLOAD_SIZE            AUTOMATIC(0) 
         PCIE_CREDIT_TOKEN_TIMEOUT           0            
         ACCURATE_TX_SCHEDULER               False(0)     
         PARTIAL_RESET_EN                    False(0)     
         SW_RECOVERY_ON_ERRORS               False(0)     
         RESET_WITH_HOST_ON_ERRORS           False(0)     
         PCI_BUS0_RESTRICT_SPEED             PCI_GEN_1(0) 
         PCI_BUS0_RESTRICT_ASPM              False(0)     
         PCI_BUS0_RESTRICT_WIDTH             PCI_X1(0)    
         PCI_BUS0_RESTRICT                   False(0)     
         CQE_COMPRESSION                     BALANCED(0)  
         IP_OVER_VXLAN_EN                    False(0)     
         MKEY_BY_NAME                        False(0)     
         UCTX_EN                             True(1)      
         PCI_ATOMIC_MODE                     PCI_ATOMIC_DISABLED_EXT_ATOMIC_ENABLED(0)
         TUNNEL_ECN_COPY_DISABLE             False(0)     
         LRO_LOG_TIMEOUT0                    6            
         LRO_LOG_TIMEOUT1                    7            
         LRO_LOG_TIMEOUT2                    8            
         LRO_LOG_TIMEOUT3                    13           
         TX_SCHEDULER_BURST                  0            
         LOG_DCR_HASH_TABLE_SIZE             14           
         DCR_LIFO_SIZE                       16384        
         ROCE_CC_PRIO_MASK_P1                255          
         CLAMP_TGT_RATE_AFTER_TIME_INC_P1    True(1)      
         CLAMP_TGT_RATE_P1                   False(0)     
         RPG_TIME_RESET_P1                   300          
         RPG_BYTE_RESET_P1                   32767        
         RPG_THRESHOLD_P1                    1            
         RPG_MAX_RATE_P1                     0            
         RPG_AI_RATE_P1                      5            
         RPG_HAI_RATE_P1                     50           
         RPG_GD_P1                           11           
         RPG_MIN_DEC_FAC_P1                  50           
         RPG_MIN_RATE_P1                     1            
         RATE_TO_SET_ON_FIRST_CNP_P1         0            
         DCE_TCP_G_P1                        1019         
         DCE_TCP_RTT_P1                      1            
         RATE_REDUCE_MONITOR_PERIOD_P1       4            
         INITIAL_ALPHA_VALUE_P1              1023         
         MIN_TIME_BETWEEN_CNPS_P1            4            
         CNP_802P_PRIO_P1                    6            
         CNP_DSCP_P1                         48           
         LLDP_NB_DCBX_P1                     False(0)     
         LLDP_NB_RX_MODE_P1                  OFF(0)       
         LLDP_NB_TX_MODE_P1                  OFF(0)       
         DCBX_IEEE_P1                        True(1)      
         DCBX_CEE_P1                         True(1)      
         DCBX_WILLING_P1                     True(1)      
         KEEP_ETH_LINK_UP_P1                 True(1)      
         KEEP_IB_LINK_UP_P1                  False(0)     
         KEEP_LINK_UP_ON_BOOT_P1             False(0)     
         KEEP_LINK_UP_ON_STANDBY_P1          False(0)     
         DO_NOT_CLEAR_PORT_STATS_P1          False(0)     
         AUTO_POWER_SAVE_LINK_DOWN_P1        False(0)     
         NUM_OF_VL_P1                        _4_VLs(3)    
         NUM_OF_TC_P1                        _8_TCs(0)    
         NUM_OF_PFC_P1                       8            
         VL15_BUFFER_SIZE_P1                 0            
         DUP_MAC_ACTION_P1                   LAST_CFG(0)  
         SRIOV_IB_ROUTING_MODE_P1            LID(1)       
         IB_ROUTING_MODE_P1                  LID(1)       
         PCI_WR_ORDERING                     per_mkey(0)  
         MULTI_PORT_VHCA_EN                  False(0)     
         PORT_OWNER                          True(1)      
         ALLOW_RD_COUNTERS                   True(1)      
         RENEG_ON_CHANGE                     True(1)      
         TRACER_ENABLE                       True(1)      
         IP_VER                              IPv4(0)      
         UEFI_HII_EN                         True(1)      
         BOOT_DBG_LOG                        False(0)     
         UEFI_LOGS                           DISABLED(0)  
         BOOT_INTERRUPT_DIS                  False(0)     
         BOOT_LACP_DIS                       True(1)      
         DYNAMIC_VF_MSIX_TABLE               False(0)     
         EXP_ROM_UEFI_ARM_ENABLE             False(0)     
         EXP_ROM_UEFI_x86_ENABLE             False(0)     
         EXP_ROM_PXE_ENABLE                  True(1)      
         ADVANCED_PCI_SETTINGS               False(0)     
         SAFE_MODE_THRESHOLD                 10           
         SAFE_MODE_ENABLE                    True(1)
Just to clarify my setup here in case anything looks wrong:
OS2 LC-LC Singlemode Duplex
KAIAM QSFP+40G-LR4 lite
Mellanox MCX4131A-GCAT_C05
 

crackelf

Member
Apr 11, 2021
74
6
8
I've used tons of different NIC vendors for a decade+ and can confidently say Mellanox is far and away the nicest to work with and the most reliable, I'd also wager it's what 90%+ users in this thread are using. I'd wager you either got bad cards, or more likely are on old or OEM-specific firmware and need to be reflashed
I updated the fw through the Mellanox MLNX_EN software on their website, but still nothing. It was one fw version out of date, so I didn't expect it to fix much. Another user is saying the fiber I have & transceivers won't work with the card:
LX4 optics are hi-power, relative to DACs or SR4 and, at least in the case of the Mellanox SX60xx series of switch, are not well supported. I struggled for years to get one to work with my SX6018, hacking away at the firmware to no avail... then one day a good samaritan PM'd me a secret cli command to enable LX4 and I was off to the races.

Unless you need duplex LC/LC, I would recommend getting a cheap SR4 transceiver and a cheap OM4 MPT cable. Or, if your equipment is close enough and bend radii aren't an issue, just go with a DAC.
Which was my original plan:
Thanks for the response and the link! I had been looking at these Brocade optics on eBay and was going to try out these Arista MTP-MTP 12 fibers OM4 (also eBay)
but was convinced otherwise:
yes they're regular 40gbe ports. instead of expensive annoying mtp you can grab these BiDi optics and run 40gbE over cheap regular singlemode duplex LC fiber XQX2502 KAIAM QSFP+40G-LR4 Lite OPTICAL MODULE NEW PULLS | eBay
 
Last edited:

crackelf

Member
Apr 11, 2021
74
6
8
I've used tons of different NIC vendors for a decade+ and can confidently say Mellanox is far and away the nicest to work with and the most reliable, I'd also wager it's what 90%+ users in this thread are using. I'd wager you either got bad cards, or more likely are on old or OEM-specific firmware and need to be reflashed
Side note: if these were so nice to work with and reliable I wouldn't be posting here asking for help. There are nearly zero resources online for these that I can find. When you search "mlx5_core + debian" my servethehome forum post is literally the 1st to 5th search result in SEO.

I truly appreciate everybody's help; if it weren't for the kind users of this forum responding to that thread I wouldn't have the virtual functions *at all*, but I can't help and feel that your brazen confidence regarding these requires a warning or a caveat emptor at the least. If you don't feel like that's the case then I am happy to have said it for myself and others that also trust your word blindly.

My experience so far with these has been plugging them in, not working, post for help on a forum, wait for someone with the knowledge-base to answer, fail again, post again, wait again, etc. I wouldn't wish troubleshooting these cards across a few weeks while a decade old switch reboots loudly every 5 minutes upon my worst enemy.
My advice to the thread: stay very far away from Mellanox if you value your time.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,869
3,372
113
34
fohdeesha.com
Another user is saying the fiber I have & transceivers won't work with the card
but was convinced otherwise
He's wrong like most people on the internet, they're not high power LR4 optics. They're LR4 Lite, and use less power than a lot of SR4 optics. On top of that, *gestures broadly to all the people in this thread using these exact optics in mellanox cards, including this post being written over one*. As mentioned previously, it's entirely possible you got bad fiber or optics, it's ebay, not amazon


There are nearly zero resources online for these that I can find. When you search "mlx5_core + debian" my servethehome forum post is literally the 1st to 5th search result in SEO.
The card was plug and play as described, you were trying to set up SR-IOV with VFs which is extra steps and kernel ****ery with any NIC vendor. Have you not set them up before? Mellanox has quite a bit of documentation on this, you linked to it yourself in the thread. Doing the same on Qlogic, broadcom, etc cards on a specific distro and setup requires just as much digging through mostly unrelated documentation, VFs and their implementation aren't standardized across distros and drivers. Again, you're free to come to any conclusion you'd like, I would just feel amiss if I didn't point out how funny it looks when you post "stay very far away from mellanox" in a thread with probably more than 100 people using mellanox NICs with *zero* issue. If you want to talk about wasting time, try crossflashing, modifying, or recovering bricked intel, qlogic, broadcom, etc NICs on boards with no rescue bootloader (one of Mellanox's best features, to be honest)
 

sko

Active Member
Jun 11, 2021
367
223
43
Side note: if these were so nice to work with and reliable I wouldn't be posting here asking for help. There are nearly zero resources online for these that I can find. When you search "mlx5_core + debian" my servethehome forum post is literally the 1st to 5th search result in SEO.

I truly appreciate everybody's help; if it weren't for the kind users of this forum responding to that thread I wouldn't have the virtual functions *at all*, but I can't help and feel that your brazen confidence regarding these requires a warning or a caveat emptor at the least. If you don't feel like that's the case then I am happy to have said it for myself and others that also trust your word blindly.

My experience so far with these has been plugging them in, not working, post for help on a forum, wait for someone with the knowledge-base to answer, fail again, post again, wait again, etc. I wouldn't wish troubleshooting these cards across a few weeks while a decade old switch reboots loudly every 5 minutes upon my worst enemy.
We're running a few Mellanox 10G adapters here /w FreeBSD and illumos/SmartOS and they are by far the most unobtrusive and easiest to work with - i.e. they 'just work'™. IIRC they all just got the cheapest (18EUR/20$) transceivers from FS and worked OOTB, so nothing special here either.
Even the tried-and-true Intel X520s once gave us a headache because a firmware update caused them to shut down the second port if they were used with 3rd party transceivers. Never saw such BS with mellanox.
 

crackelf

Member
Apr 11, 2021
74
6
8
We're running a few Mellanox 10G adapters here /w FreeBSD and illumos/SmartOS and they are by far the most unobtrusive and easiest to work with - i.e. they 'just work'™. IIRC they all just got the cheapest (18EUR/20$) transceivers from FS and worked OOTB, so nothing special here either.
Even the tried-and-true Intel X520s once gave us a headache because a firmware update caused them to shut down the second port if they were used with 3rd party transceivers. Never saw such BS with mellanox.
I'm happy they work for you. They don't work for me. It's almost as if life is subjective.
 

crackelf

Member
Apr 11, 2021
74
6
8
He's wrong like most people on the internet
So you're infallible? To me everyone here is just text on a screen. It's all equal in my eyes until I see something functional.
they're not high power LR4 optics. They're LR4 Lite, and use less power than a lot of SR4 optics. On top of that, *gestures broadly to all the people in this thread using these exact optics in mellanox cards, including this post being written over one*. As mentioned previously, it's entirely possible you got bad fiber or optics, it's ebay, not amazon
Happy to be proven otherwise, but I've tried 3 different fiber cables and 20+ different KAIAM transceivers. I'm getting some DAC's and some OS2 soon to test.
The card was plug and play as described
Are they though? I literally plugged two random ones out of the 12 in front of me fresh out of the box and they connect to each other at 1G.
you were trying to set up SR-IOV with VFs which is extra steps and kernel ****ery with any NIC vendor.
With or without SR-IOV enabled they still don't connect... Stock from the box they're 1G.
Have you not set them up before? Mellanox has quite a bit of documentation on this, you linked to it yourself in the thread.
First time with Mellanox. 100% of the Mellanox documentation I found was either incorrect, out of date, or for the wrong card entirely. Their driver page for this card literally links out incorrectly to an entirely different ConnectX. I had to HTML crawl through their directories. Can you imagine if Intel Ark did that?
Doing the same on Qlogic, broadcom, etc cards on a specific distro and setup requires just as much digging through mostly unrelated documentation, VFs and their implementation aren't standardized across distros and drivers.
Exactly why I don't go anywhere near Qlogic or broadcom.
Again, you're free to come to any conclusion you'd like, I would just feel amiss if I didn't point out how funny it looks when you post "stay very far away from mellanox" in a thread with probably more than 100 people using mellanox NICs with *zero* issue.
So why can none of the more than 100 people figure out what's wrong? I'm just pointing out the contradiction. I'm genuinely confused and nobody can tell me what exactly is the issue here.

My goal is simply publishing that I had the very opposite experience. In my opinion it is disingenuous to say anything near "they just work" if this is a potential outcome. There's a lot of bravado in this thread if that's the case.
If you want to talk about wasting time, try crossflashing, modifying, or recovering bricked intel, qlogic, broadcom, etc NICs on boards with no rescue bootloader (one of Mellanox's best features, to be honest)
I'll leave that to the pros. I'm not saying my strife is greater than anybody else's.