hmm maybe try removing the "stack disable" option, (either "no stack disable" or "stack enable" can't remember which) then rebooting. then if you can paste the output of (assuming 1/3/1 is actually plugged into a card):
show int e 1/3/1
show media ethernet 1/3/1
show optic 1/3/1
Update: these Connectx-4 LX are running @1Gbps when connected to each other, and I can't get them to connect even directly with anything higher than that. My advice to the thread: stay very far away from Mellanox if you value your time.
Got a small problem where I can't get the 10GB breakout interfaces to come up. I'm trying to brake out the 40GB to 10GB LCs using some FS.com gear. I've followed the setup guide for removing the ports from the stack.
1/3/7 is an optic chucked into the front port used to test the LC cables. The other 1/3's are LAGs go to other switchgear.
Code:
SSH@sw-core(config)#show conf
!
Startup-config data location is flash memory
!
Startup configuration:
!
ver 08.0.30uT7f3
!
stack unit 1
module 1 icx6610-48p-poe-port-management-module
module 2 icx6610-qsfp-10-port-160g-module
module 3 icx6610-8-port-10g-dual-mode-module
stack disable
FWIW, I set the 1/2/2 to 1/2/5 and 1/2/7 to 1/2/10 to speed-duplex 10G-full also left blank.
Here's the interface summary
Code:
1/2/1 Down None None None None Yes 1 0 748e.f8fe.c148
1/2/2 Down None None None None No 1 0 748e.f8fe.c148
1/2/3 Down None None None None Yes 1 0 748e.f8fe.c148
1/2/4 Down None None None None No 1 0 748e.f8fe.c148
1/2/5 Down None None None None No 1 0 748e.f8fe.c148
1/2/6 Down None None None None No 1 0 748e.f8fe.c148
1/2/7 Down None None None None Yes 1 0 748e.f8fe.c148 SW-XXX1
1/2/8 Down None None None None Yes 1 0 748e.f8fe.c148 SW-XXX2
1/2/9 Down None None None None Yes 1 0 748e.f8fe.c148 SW-XXX3
1/2/10 Down None None None None Yes 1 0 748e.f8fe.c148
1/3/1 Down None None None None Yes 1 0 748e.f8fe.c148
1/3/2 Down None None None None Yes 1 0 748e.f8fe.c148
1/3/3 Up Forward Full 10G 3 Yes 1 0 748e.f8fe.c148 SW-DXX-1
1/3/4 Up Forward Full 10G 3 Yes 1 0 748e.f8fe.c148 SW-DXX-2
1/3/5 Up Forward Full 10G 2 Yes 1 0 748e.f8fe.c148 SW-RXX-1
1/3/6 Up Forward Full 10G 2 Yes 1 0 748e.f8fe.c148 SW-RXX-2
1/3/7 Down None None None None Yes 1 0 748e.f8fe.c148
1/3/8 Down None None None None Yes 1 0 748e.f8fe.c148
Here's the interface that has a plugged in LC cable:
Code:
SSH@sw-core(config)#show int ethe 1/2/7
10GigabitEthernet 1/2/7 is down, line protocol is down
Port down for 2 hour(s) 25 minute(s) 13 second(s)
Hardware is 10GigabitEthernet , address is 748e.f8fe.c148 (bia 748e.f8fe.c17f)
Configured speed 10Gbit, actual unknown, configured duplex fdx, actual unknown
Configured mdi mode AUTO, actual unknown
Member of 11 L2 VLANs, port is dual mode in Vlan 1, port state is BLOCKING
BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
Link Error Dampening is Disabled
STP configured to ON, priority is level0, mac-learning is enabled
Openflow is Disabled, Openflow Hybrid mode is Disabled, Flow Control is config enabled, oper enabled, negotiation disabled
Mirror disabled, Monitor disabled
Mac-notification is disabled
Not member of any active trunks
Not member of any configured trunks
Port name is SW-XXX1
MTU 10200 bytes, encapsulation ethernet
300 second input rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
300 second output rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
54035 packets input, 15271970 bytes, 0 no buffer
Received 2649 broadcasts, 3205 multicasts, 48181 unicasts
0 input errors, 0 CRC, 0 frame, 0 ignored
0 runts, 0 giants
129068 packets output, 41755954 bytes, 0 underruns
Transmitted 11005 broadcasts, 55975 multicasts, 62088 unicasts
0 output errors, 0 collisions
Relay Agent Information option: Disabled
Egress queues:
Queue counters Queued packets Dropped Packets
0 115303 0
1 0 0
2 0 0
3 0 0
4 0 0
5 1 0
6 0 0
Code:
SSH@sw-core>show ver
Copyright (c) 1996-2016 Brocade Communications Systems, Inc. All rights reserved.
UNIT 1: compiled on Apr 23 2020 at 13:17:12 labeled as FCXR08030u
(10545591 bytes) from Primary FCXR08030u.bin
SW: Version 08.0.30uT7f3
Boot-Monitor Image size = 370695, Version:10.1.00T7f5 (grz10100)
HW: Stackable ICX6610-48-HPOE
==========================================================================
UNIT 1: SL 1: ICX6610-48P POE 48-port Management Module
Serial #: 2ax5o2jk68e
License: ICX6610_ADV_ROUTER_SOFT_PACKAGE (LID: H4CKTH3PLN8)
P-ENGINE 0: type E02B, rev 01
P-ENGINE 1: type E02B, rev 01
==========================================================================
UNIT 1: SL 2: ICX6610-QSFP 10-port 160G Module
==========================================================================
UNIT 1: SL 3: ICX6610-8-port Dual Mode(SFP/SFP+) Module
==========================================================================
800 MHz Power PC processor 8544E (version 0021/0023) 400 MHz bus
65536 KB flash memory
512 MB DRAM
STACKID 1 system uptime is 5 day(s) 1 hour(s) 38 minute(s) 27 second(s)
The system started at 12:06:39 GMT+10 Sun Feb 20 2022
The system : started=cold start
I've double checked that the cables are plugged in. I have 2 x of the above kit and I get the same behaviour on each 40GQSFP, MTP cable and breakout cassette.
I just picked up a secondhand 7250-48p with the help of the information in this thread. Thank you.
I am having an issue that POE does not work on some ports and I'm trying to figure out if there is a hardware issue or a software issue.
I followed the guide to get the switch updated and everything looked as I expected. Then I updated to the latest version 9.0.10a.
When I run show inline power detail debug-info I see (6) modules with HW version V1R3. However one module shows Vm2vErr as status and five show VOP-Sev1 as status. Only one is in Good status.
EDIT: I confirmed that the first 8 ports that are in Vm2vErr are the ports that are not powering POE devices.
Does anyone have any reference to these status codes and/or knows how to troubleshoot this? Google search turns up almost nothing.
FastIron Command Reference Guide has this information but I am not sure what causes these statuses or how to correct them.
The status of the device:
• Failed: No (bad or unreachable) PoE device found (bootup time)
• Good: OK - Expected PoE device Found (Zone1, no VOP)
• Revived: Device is currently refreshed
• n/a: Reserved
• Lost: Device lost or different from expected (while in operation)
• VOP-Sev1: Device error 1 (Zone2, VOP severity 1)
• VOP-Sev2: Device error 1 (Zone2, VOP severity 2)
• VOP-Sev3: Device error 1 (Zone2, VOP severity 3)
• VmErr: Device Vmain Error
• Vm2vErr: Device Vmain < System AVG Vmain by 2v or more
• Recovered: OK - Expected PoE device Found (Zone3, recovered from VOP)
Full output of show inline power detail debug-info:
Power Supply Data On unit 1:
++++++++++++++++++
Power Supply Data:
++++++++++++++++++
Power Supply #1:
Max Curr: 13.7 Amps
Voltage: 54.0 Volts
Capacity: 740 Watts
PoePower: 740 Watts
power supply 2 is not present
power supply 3 is not present
POE Details Info. On Unit 1 :
General PoE Data:
+++++++++++++++++
Firmware
Version
----------------
02.1.8 Build 004
Cumulative Port Power Data:
+++++++++++++++++++++++++++
#Ports #Ports #Ports Power Power
Pri: 1 Pri: 2 Pri: 3 Consumption Allocation
-----------------------------------------------
0 0 48 8.400 W 44.932 W
show version
Copyright (c) Ruckus Networks, Inc. All rights reserved.
UNIT 1: compiled on Feb 8 2022 at 22:14:30 labeled as SPR09010a
(33554432 bytes) from Primary SPR09010a.bin (UFI)
SW: Version 09.0.10aT213
Compressed Primary Boot Code size = 786944, Version:10.1.22T215 (spz10122)
Compiled on Mon Jan 24 06:37:29 2022
HW: Stackable ICX7250-48-HPOE
==========================================================================
UNIT 1: SL 1: ICX7250-48P POE 48-port Management Module
Serial #UK384XXXXX
Software Package: ICX7250_L3_SOFT_PACKAGE
Current License: l3-prem-8X10G
P-ASIC 0: type B344, rev 01 Chip BCM56344_A0
==========================================================================
UNIT 1: SL 2: ICX7250-SFP-Plus 8-port 80G Module
==========================================================================
1000 MHz ARM processor ARMv7 88 MHz bus
8 MB boot flash memory
2 GB code flash memory
2 GB DRAM
STACKID 1 system uptime is 44 minute(s) 11 second(s)
The system started at 19:41:03 GMT-05 Mon Feb 28 2022
The system : started=warm start reloaded=by "reload"
show chassis
The stack unit 1 chassis info:
Power supply 1 (AC - PoE) present, status ok
Power supply 2 not present
Power supply 3 not present
Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 ok, speed (auto): [[1]]<->2
Fan 3 ok, speed (auto): [[1]]<->2
Fan controlled temperature:
Rule 1/2 (MGMT THERMAL PLANE): 66.3 deg-C
Rule 2/2 (AIR OUTLET NEAR PSU): 23.5 deg-C
Fan 1 Air Flow Direction: Front to Back
Fan 2 Air Flow Direction: Front to Back
Fan 3 Air Flow Direction: Front to Back
Slot 1 Current Temperature: 66.3 deg-C (Sensor 1), 24.5 deg-C (Sensor 2)
Slot 2 Current Temperature: NA
Warning level.......: 100.0 deg-C
Shutdown level......: 105.0 deg-C
I just picked up a secondhand 7250-48p with the help of the information in this thread. Thank you.
I am having an issue that POE does not work on some ports and I'm trying to figure out if there is a hardware issue or a software issue.
I followed the guide to get the switch updated and everything looked as I expected. Then I updated to the latest version 9.0.10a.
When I run show inline power detail debug-info I see (6) modules with HW version V1R3. However one module shows Vm2vErr as status and five show VOP-Sev1 as status. Only one is in Good status.
EDIT: I confirmed that the first 8 ports that are in Vm2vErr are the ports that are not powering POE devices.
Does anyone have any reference to these status codes and/or knows how to troubleshoot this? Google search turns up almost nothing.
FastIron Command Reference Guide has this information but I am not sure what causes these statuses or how to correct them.
The status of the device:
• Failed: No (bad or unreachable) PoE device found (bootup time)
• Good: OK - Expected PoE device Found (Zone1, no VOP)
• Revived: Device is currently refreshed
• n/a: Reserved
• Lost: Device lost or different from expected (while in operation)
• VOP-Sev1: Device error 1 (Zone2, VOP severity 1)
• VOP-Sev2: Device error 1 (Zone2, VOP severity 2)
• VOP-Sev3: Device error 1 (Zone2, VOP severity 3)
• VmErr: Device Vmain Error
• Vm2vErr: Device Vmain < System AVG Vmain by 2v or more
• Recovered: OK - Expected PoE device Found (Zone3, recovered from VOP)
Full output of show inline power detail debug-info:
Power Supply Data On unit 1:
++++++++++++++++++
Power Supply Data:
++++++++++++++++++
Power Supply #1:
Max Curr: 13.7 Amps
Voltage: 54.0 Volts
Capacity: 740 Watts
PoePower: 740 Watts
power supply 2 is not present
power supply 3 is not present
POE Details Info. On Unit 1 :
General PoE Data:
+++++++++++++++++
Firmware
Version
----------------
02.1.8 Build 004
Cumulative Port Power Data:
+++++++++++++++++++++++++++
#Ports #Ports #Ports Power Power
Pri: 1 Pri: 2 Pri: 3 Consumption Allocation
-----------------------------------------------
0 0 48 8.400 W 44.932 W
show version
Copyright (c) Ruckus Networks, Inc. All rights reserved.
UNIT 1: compiled on Feb 8 2022 at 22:14:30 labeled as SPR09010a
(33554432 bytes) from Primary SPR09010a.bin (UFI)
SW: Version 09.0.10aT213
Compressed Primary Boot Code size = 786944, Version:10.1.22T215 (spz10122)
Compiled on Mon Jan 24 06:37:29 2022
HW: Stackable ICX7250-48-HPOE
==========================================================================
UNIT 1: SL 1: ICX7250-48P POE 48-port Management Module
Serial #UK384XXXXX
Software Package: ICX7250_L3_SOFT_PACKAGE
Current License: l3-prem-8X10G
P-ASIC 0: type B344, rev 01 Chip BCM56344_A0
==========================================================================
UNIT 1: SL 2: ICX7250-SFP-Plus 8-port 80G Module
==========================================================================
1000 MHz ARM processor ARMv7 88 MHz bus
8 MB boot flash memory
2 GB code flash memory
2 GB DRAM
STACKID 1 system uptime is 44 minute(s) 11 second(s)
The system started at 19:41:03 GMT-05 Mon Feb 28 2022
The system : started=warm start reloaded=by "reload"
show chassis
The stack unit 1 chassis info:
Power supply 1 (AC - PoE) present, status ok
Power supply 2 not present
Power supply 3 not present
Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 ok, speed (auto): [[1]]<->2
Fan 3 ok, speed (auto): [[1]]<->2
Fan controlled temperature:
Rule 1/2 (MGMT THERMAL PLANE): 66.3 deg-C
Rule 2/2 (AIR OUTLET NEAR PSU): 23.5 deg-C
Fan 1 Air Flow Direction: Front to Back
Fan 2 Air Flow Direction: Front to Back
Fan 3 Air Flow Direction: Front to Back
Slot 1 Current Temperature: 66.3 deg-C (Sensor 1), 24.5 deg-C (Sensor 2)
Slot 2 Current Temperature: NA
Warning level.......: 100.0 deg-C
Shutdown level......: 105.0 deg-C
Update: these Connectx-4 LX are running @1Gbps when connected to each other, and I can't get them to connect even directly with anything higher than that. My advice to the thread: stay very far away from Mellanox if you value your time.
I've used tons of different NIC vendors for a decade+ and can confidently say Mellanox is far and away the nicest to work with and the most reliable, I'd also wager it's what 90%+ users in this thread are using. I'd wager you either got bad cards, or more likely are on old or OEM-specific firmware and need to be reflashed
Update: these Connectx-4 LX are running @1Gbps when connected to each other, and I can't get them to connect even directly with anything higher than that. My advice to the thread: stay very far away from Mellanox if you value your time.
In case someone (like me) is searching for a comparison of the ICX 7250-48p noise level to cisco 3750X:
The 7250-48P is _a lot_ louder than a stack of 2 3750X (-48p & -12s, both /w NM-10G module) with 4 fans and 4 750W PSUs! The noise of the brocade is also much higher pitched, and therefore more present and annoying than the rather muffled rushing of the ciscos.
It also runs WAY hotter than the ciscos even if it was just idling vs the cisco stack under normal office loads.
I first compared it to my single 3750X-24P at home and was quite shocked that this is considered "almost silent" in brocade-terms. I then took it to the office where it now sat besides the (open) network cabinet with the cisco stack inside and it is definitely too loud to go anywhere someone can hear it.
So for (home)-office or homelab-use if you are in the same situation: just stay with the 3750X, even if you have to stack several of them they will still be quieter.
I wanted to upgrade from the 3750x-24p /w C3KX-NM-10G to the 7250-48P because it looked very promising and intriguing on paper, but I will now just add another 3750x with 10G module, so I will at least have 4x10G, and then I'll just wait another year or so until the 3850s have dropped in price a little more (those are even a bit quieter than the 3750x!).
In case someone (like me) is searching for a comparison of the ICX 7250-48p noise level to cisco 3750X:
The 7250-48P is _a lot_ louder than a stack of 2 3750X (-48p & -12s, both /w NM-10G module) with 4 fans and 4 750W PSUs! The noise of the brocade is also much higher pitched, and therefore more present and annoying than the rather muffled rushing of the ciscos.
It also runs WAY hotter than the ciscos even if it was just idling vs the cisco stack under normal office loads.
I first compared it to my single 3750X-24P at home and was quite shocked that this is considered "almost silent" in brocade-terms. I then took it to the office where it now sat besides the (open) network cabinet with the cisco stack inside and it is definitely too loud to go anywhere someone can hear it.
So for (home)-office or homelab-use if you are in the same situation: just stay with the 3750X, even if you have to stack several of them they will still be quieter.
I wanted to upgrade from the 3750x-24p /w C3KX-NM-10G to the 7250-48P because it looked very promising and intriguing on paper, but I will now just add another 3750x with 10G module, so I will at least have 4x10G, and then I'll just wait another year or so until the 3850s have dropped in price a little more (those are even a bit quieter than the 3750x!).
the 48-port PoE version of the 7250 has 3 fans, versus 1 for the regular version. As covered previously in the thread, you can unplug all but 1 fan with no ill effects assuming you're not running tens of high powered PoE devices. Of course it runs hotter than your cisco, it has twice the ports (twice the ASIC), and twice the PoE per chassis volume. lastly, if it's that loud, the fan(s) in it might be on their way out with failing bearings, especially if it's "high pitched" like you say. I have an ICX7250-24 on the desk behind me and I literally cannot hear it over the laptop on top of it
the 48-port PoE version of the 7250 has 3 fans, versus 1 for the regular version. As covered previously in the thread, you can unplug all but 1 fan with no ill effects assuming you're not running tens of high powered PoE devices. Of course it runs hotter than your cisco, it has twice the ports (twice the ASIC), and twice the PoE per chassis volume lastly, if it's that loud, the fan(s) in it might be on their way out with failing bearings, especially if it's "high pitched" like you say. I have an ICX7250-24 on the desk behind me and I literally cannot hear it over the laptop on top of it
Thanks for the input. I already tried unplugging one or more fans, which doesn't do much about the noise, but then the temperature rises within the first ~20 minutes above the threshold for fan speed 2 - i.e.: it gets even louder.
The comparisons (noise and temperatures) I mentioned were against the stack at the office, not the single -24p at home.
I am aware this switch COULD push a lot more packets than the 3750X, but at idle IMHO it should still be able to be as quiet as a stack with 48port PoE + 12 port SFP with 4 10GBit ports utilized under load.
The fans sound healthy and not like their bearings are dying, but I still might try to get some replacement fans. I've already searched for some options, but neither one of the Gryphoon or Deltas mentioned here are available here in germany/eu, let alone for a reasonable price from overseas (~40EUR+ / piece and even more for shipping sadly isn't "reasonable" - this would be over 50% of what the switch had cost...). I also searched for comparable models from Sunon or SanyoDenki, but except for OEM-volume-orders, especially the Sanys are next to impossible to get, or insanely expensive in single-digit quantities.
I already tried some Deltas, Sanyo Denki or Sunon I had lying around from servers or switches, but either the switch gets way too hot (and spins up the fans to even more annoying levels) or it won't be (much) quieter.
One major factor besides the fans seems to be the rather thin sheet metal that is used for the case - the top seems to resonate and amplify the fan noise, which gets better if some weight is applied. Also there is absolutely no guidance of the airflow, so most of the air seems to go from the PSU side straight to the fans, instead of being pulled over the ASICs and PoE board at the front of the switch. Don't know if this is normal or if there is some shroud missing inside of mine...
To be clear: given the raw performance data this thing still is dirt-cheap and "reasonably quiet" for something that should usually go into a server-room in the basement or in a datacenter.
But I think if noise is a point and/or you don't need this amount of throughput and *especially* if you already have one or more 3750X running, it isn't worth switching to the 7250 unless e.g. you absolutely need that amount of 10G ports (or the 40G ports on the 6610).
I too was a little surprised at how loud the 7250-48p switch is that I just received. I expected it to be quieter based on the spec sheet and the information posted in this thread.
I could not sit with the switch in the same room as me. It's fine in my basement though.
In my case it'll be going back to the seller as my switch has a hardware failure with POE (unless the seller lets me keep it for cheap).
Thanks for the input. I already tried unplugging one or more fans, which doesn't do much about the noise, but then the temperature rises within the first ~20 minutes above the threshold for fan speed 2 - i.e.: it gets even louder.
The comparisons (noise and temperatures) I mentioned were against the stack at the office, not the single -24p at home.
I am aware this switch COULD push a lot more packets than the 3750X, but at idle IMHO it should still be able to be as quiet as a stack with 48port PoE + 12 port SFP with 4 10GBit ports utilized under load.
The fans sound healthy and not like their bearings are dying, but I still might try to get some replacement fans. I've already searched for some options, but neither one of the Gryphoon or Deltas mentioned here are available here in germany/eu, let alone for a reasonable price from overseas (~40EUR+ / piece and even more for shipping sadly isn't "reasonable" - this would be over 50% of what the switch had cost...). I also searched for comparable models from Sunon or SanyoDenki, but except for OEM-volume-orders, especially the Sanys are next to impossible to get, or insanely expensive in single-digit quantities.
I already tried some Deltas, Sanyo Denki or Sunon I had lying around from servers or switches, but either the switch gets way too hot (and spins up the fans to even more annoying levels) or it won't be (much) quieter.
One major factor besides the fans seems to be the rather thin sheet metal that is used for the case - the top seems to resonate and amplify the fan noise, which gets better if some weight is applied. Also there is absolutely no guidance of the airflow, so most of the air seems to go from the PSU side straight to the fans, instead of being pulled over the ASICs and PoE board at the front of the switch. Don't know if this is normal or if there is some shroud missing inside of mine...
To be clear: given the raw performance data this thing still is dirt-cheap and "reasonably quiet" for something that should usually go into a server-room in the basement or in a datacenter.
But I think if noise is a point and/or you don't need this amount of throughput and *especially* if you already have one or more 3750X running, it isn't worth switching to the 7250 unless e.g. you absolutely need that amount of 10G ports (or the 40G ports on the 6610).
No link type on this card. Already got the VF's working and loaded in Debian, but still nothing. I suspect the fiber or transceiver don't work with this card, despite advice.
Find many great new & used options and get the best deals for XQX2502 KAIAM QSFP+40G-LR4 Lite OPTICAL MODULE NEW PULLS at the best online prices at eBay! Free shipping for many products!
Find many great new & used options and get the best deals for Mellanox MCX4131A-GCAT_C05 ConnectX-4 LX 50GbE PCIe Network Card Newest Firmware at the best online prices at eBay! Free shipping for many products!
I've used tons of different NIC vendors for a decade+ and can confidently say Mellanox is far and away the nicest to work with and the most reliable, I'd also wager it's what 90%+ users in this thread are using. I'd wager you either got bad cards, or more likely are on old or OEM-specific firmware and need to be reflashed
I updated the fw through the Mellanox MLNX_EN software on their website, but still nothing. It was one fw version out of date, so I didn't expect it to fix much. Another user is saying the fiber I have & transceivers won't work with the card:
LX4 optics are hi-power, relative to DACs or SR4 and, at least in the case of the Mellanox SX60xx series of switch, are not well supported. I struggled for years to get one to work with my SX6018, hacking away at the firmware to no avail... then one day a good samaritan PM'd me a secret cli command to enable LX4 and I was off to the races.
Unless you need duplex LC/LC, I would recommend getting a cheap SR4 transceiver and a cheap OM4 MPT cable. Or, if your equipment is close enough and bend radii aren't an issue, just go with a DAC.
I've used tons of different NIC vendors for a decade+ and can confidently say Mellanox is far and away the nicest to work with and the most reliable, I'd also wager it's what 90%+ users in this thread are using. I'd wager you either got bad cards, or more likely are on old or OEM-specific firmware and need to be reflashed
Side note: if these were so nice to work with and reliable I wouldn't be posting here asking for help. There are nearly zero resources online for these that I can find. When you search "mlx5_core + debian" my servethehome forum post is literally the 1st to 5th search result in SEO.
I truly appreciate everybody's help; if it weren't for the kind users of this forum responding to that thread I wouldn't have the virtual functions *at all*, but I can't help and feel that your brazen confidence regarding these requires a warning or a caveat emptor at the least. If you don't feel like that's the case then I am happy to have said it for myself and others that also trust your word blindly.
My experience so far with these has been plugging them in, not working, post for help on a forum, wait for someone with the knowledge-base to answer, fail again, post again, wait again, etc. I wouldn't wish troubleshooting these cards across a few weeks while a decade old switch reboots loudly every 5 minutes upon my worst enemy.
He's wrong like most people on the internet, they're not high power LR4 optics. They're LR4 Lite, and use less power than a lot of SR4 optics. On top of that, *gestures broadly to all the people in this thread using these exact optics in mellanox cards, including this post being written over one*. As mentioned previously, it's entirely possible you got bad fiber or optics, it's ebay, not amazon
There are nearly zero resources online for these that I can find. When you search "mlx5_core + debian" my servethehome forum post is literally the 1st to 5th search result in SEO.
The card was plug and play as described, you were trying to set up SR-IOV with VFs which is extra steps and kernel ****ery with any NIC vendor. Have you not set them up before? Mellanox has quite a bit of documentation on this, you linked to it yourself in the thread. Doing the same on Qlogic, broadcom, etc cards on a specific distro and setup requires just as much digging through mostly unrelated documentation, VFs and their implementation aren't standardized across distros and drivers. Again, you're free to come to any conclusion you'd like, I would just feel amiss if I didn't point out how funny it looks when you post "stay very far away from mellanox" in a thread with probably more than 100 people using mellanox NICs with *zero* issue. If you want to talk about wasting time, try crossflashing, modifying, or recovering bricked intel, qlogic, broadcom, etc NICs on boards with no rescue bootloader (one of Mellanox's best features, to be honest)
Side note: if these were so nice to work with and reliable I wouldn't be posting here asking for help. There are nearly zero resources online for these that I can find. When you search "mlx5_core + debian" my servethehome forum post is literally the 1st to 5th search result in SEO.
I truly appreciate everybody's help; if it weren't for the kind users of this forum responding to that thread I wouldn't have the virtual functions *at all*, but I can't help and feel that your brazen confidence regarding these requires a warning or a caveat emptor at the least. If you don't feel like that's the case then I am happy to have said it for myself and others that also trust your word blindly.
My experience so far with these has been plugging them in, not working, post for help on a forum, wait for someone with the knowledge-base to answer, fail again, post again, wait again, etc. I wouldn't wish troubleshooting these cards across a few weeks while a decade old switch reboots loudly every 5 minutes upon my worst enemy.
We're running a few Mellanox 10G adapters here /w FreeBSD and illumos/SmartOS and they are by far the most unobtrusive and easiest to work with - i.e. they 'just work'™. IIRC they all just got the cheapest (18EUR/20$) transceivers from FS and worked OOTB, so nothing special here either.
Even the tried-and-true Intel X520s once gave us a headache because a firmware update caused them to shut down the second port if they were used with 3rd party transceivers. Never saw such BS with mellanox.
We're running a few Mellanox 10G adapters here /w FreeBSD and illumos/SmartOS and they are by far the most unobtrusive and easiest to work with - i.e. they 'just work'™. IIRC they all just got the cheapest (18EUR/20$) transceivers from FS and worked OOTB, so nothing special here either.
Even the tried-and-true Intel X520s once gave us a headache because a firmware update caused them to shut down the second port if they were used with 3rd party transceivers. Never saw such BS with mellanox.
they're not high power LR4 optics. They're LR4 Lite, and use less power than a lot of SR4 optics. On top of that, *gestures broadly to all the people in this thread using these exact optics in mellanox cards, including this post being written over one*. As mentioned previously, it's entirely possible you got bad fiber or optics, it's ebay, not amazon
Happy to be proven otherwise, but I've tried 3 different fiber cables and 20+ different KAIAM transceivers. I'm getting some DAC's and some OS2 soon to test.
First time with Mellanox. 100% of the Mellanox documentation I found was either incorrect, out of date, or for the wrong card entirely. Their driver page for this card literally links out incorrectly to an entirely different ConnectX. I had to HTML crawl through their directories. Can you imagine if Intel Ark did that?
Doing the same on Qlogic, broadcom, etc cards on a specific distro and setup requires just as much digging through mostly unrelated documentation, VFs and their implementation aren't standardized across distros and drivers.
Again, you're free to come to any conclusion you'd like, I would just feel amiss if I didn't point out how funny it looks when you post "stay very far away from mellanox" in a thread with probably more than 100 people using mellanox NICs with *zero* issue.
So why can none of the more than 100 people figure out what's wrong? I'm just pointing out the contradiction. I'm genuinely confused and nobody can tell me what exactly is the issue here.
My goal is simply publishing that I had the very opposite experience. In my opinion it is disingenuous to say anything near "they just work" if this is a potential outcome. There's a lot of bravado in this thread if that's the case.
If you want to talk about wasting time, try crossflashing, modifying, or recovering bricked intel, qlogic, broadcom, etc NICs on boards with no rescue bootloader (one of Mellanox's best features, to be honest)
This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
By continuing to use this site, you are consenting to our use of cookies.