Flashing stock Mellanox firmware to OEM (EMC) ConnectX-3 IB/Ethernet dual-port QSFP adapter

klui

Active Member
Feb 3, 2019
545
240
43
And yet, I can get them on eBay for a pack of 5 at $10.
I'm not saying one should buy those patch cords for $20 but I'm not about to buy no-names from eBay @5 for $10 either. Unless they're something like a clearance and have a Commscope or equivalent SKU.
 

klui

Active Member
Feb 3, 2019
545
240
43
Thanks! I did not know it. Is there anything else I should know about this MPO/MPT standard? Does the direction matter? How long is the shortest cable possible I could use between keystone modules? What is the minimum bending radius for such cables? Are there flexible ones?
Yes. 1m. Depends on cable type. Yes.

Look for the fiber articles on the main site.
 
  • Like
Reactions: inf3rno

LodeRunner

Active Member
Apr 27, 2019
383
158
43
Thanks! I did not know it. Is there anything else I should know about this MPO/MPT standard? Does the direction matter? How long is the shortest cable possible I could use between keystone modules? What is the minimum bending radius for such cables? Are there flexible ones?
Directionality: there are key-up and key-down connectors; as klui notes, there is at least one article on the main site about MPO/MTP.
Shortest length: I would assume this is dependent on the transceiver rather than the fiber or connector type. Cisco for example states a maximum Tx/Rx of 0.5 dB which means there's practically no minimum for some of their -LR optics. Conversely, the -ER and -ZR optics with ranges measured in 10's of km, will often require attenuators for anything shorter than half the maximum range.
Bend radius: There is fiber rated as bend-insensitive, which really means a minimum radius of 10mm I think? May be MFR dependent.
 

inf3rno

New Member
Mar 24, 2021
20
1
3
Directionality: there are key-up and key-down connectors; as klui notes, there is at least one article on the main site about MPO/MTP.
Shortest length: I would assume this is dependent on the transceiver rather than the fiber or connector type. Cisco for example states a maximum Tx/Rx of 0.5 dB which means there's practically no minimum for some of their -LR optics. Conversely, the -ER and -ZR optics with ranges measured in 10's of km, will often require attenuators for anything shorter than half the maximum range.
Bend radius: There is fiber rated as bend-insensitive, which really means a minimum radius of 10mm I think? May be MFR dependent.
Thanks! I'll buy Mellanox MC2207411-SR4L or Finisar FTL414QB2C transceiver. Most probably the latter, because it is good for longer than 30m distance too. I need this short distance only temporarily, but if it is not possible, then I'll solve it with a single longer cable and a cable channel instead of using a keystone wall plate.
 

Znuff

New Member
May 17, 2022
2
0
1
Romania
Hi,

I've followed the instructions and successfully flashed some:

ConnectX-3 VPI adapter card; dual-port QSFP; QDR IB (40Gb/s) and 10GigE; PCIe3.0x8 8GT/s; RoHS R6

to generic firmware.

And for most of it, the cards seem to work fine.

I can get around >30Gbit/s with a single iperf3 between 2 servers with these cards, which seems great.

But once I actually put them to real work, they'll cap at 17Gbit/s:

1652823039217.png

At this point, iperf3 will also return poor speeds (sub 10Mbit), so it's clear that it has reached it's capped.

Tranceivers are Arista:

Code:
Identifier                                : 0x0d (QSFP+)
Extended identifier                       : 0xc0
Extended identifier description           : 3.5W max. Power consumption
Extended identifier description           : No CDR in TX, No CDR in RX
Extended identifier description           : High Power Class (> 3.5 W) not enabled
Connector                                 : 0x07 (LC)
Transceiver codes                         : 0x02 0x00 0x00 0x00 0x10 0x10 0x01 0xd5
Transceiver type                          : 40G Ethernet: 40G Base-LR4
Transceiver type                          : FC: long distance (L)
Transceiver type                          : FC: Longwave laser (LL)
Transceiver type                          : FC: Single Mode (SM)
Transceiver type                          : FC: 1200 MBytes/sec
Transceiver type                          : FC: 800 MBytes/sec
Transceiver type                          : FC: 400 MBytes/sec
Transceiver type                          : FC: 200 MBytes/sec
Transceiver type                          : FC: 100 MBytes/sec
Encoding                                  : 0x03 (NRZ)
BR, Nominal                               : 10300Mbps
Rate identifier                           : 0x00
Length (SMF,km)                           : 10km
Length (OM3 50um)                         : 0m
Length (OM2 50um)                         : 0m
Length (OM1 62.5um)                       : 0m
Length (Copper or Active cable)           : 0m
Transmitter technology                    : 0x40 (1310 nm DFB)
Laser wavelength                          : 1310.000nm
Laser wavelength tolerance                : 10.000nm
Vendor name                               : Arista Networks
Vendor OUI                                : 64:9d:99
Vendor PN                                 : QSFP-LR4-40G
Vendor rev                                : 1A
Vendor SN                                 : F2020105366
Revision Compliance                       : SFF-8636 Rev 2.5/2.6/2.7
Module temperature                        : 39.28 degrees C / 102.70 degrees F
Module voltage                            : 3.1662 V
Alarm/warning flags implemented           : Yes
Laser tx bias current (Channel 1)         : 38.464 mA
Laser tx bias current (Channel 2)         : 40.896 mA
Laser tx bias current (Channel 3)         : 37.952 mA
Laser tx bias current (Channel 4)         : 36.864 mA
Transmit avg optical power (Channel 1)    : 0.8780 mW / -0.57 dBm
Transmit avg optical power (Channel 2)    : 0.8186 mW / -0.87 dBm
Transmit avg optical power (Channel 3)    : 1.0237 mW / 0.10 dBm
Transmit avg optical power (Channel 4)    : 0.8855 mW / -0.53 dBm
Rcvr signal avg optical power(Channel 1)  : 0.6321 mW / -1.99 dBm
Rcvr signal avg optical power(Channel 2)  : 1.0105 mW / 0.05 dBm
Rcvr signal avg optical power(Channel 3)  : 0.9729 mW / -0.12 dBm
Rcvr signal avg optical power(Channel 4)  : 0.9047 mW / -0.43 dBm
And they seem to be in fine shape.

Has anyone encountered a similar issue? I'm puzzled of what it could be.
 

ectoplasmosis

Active Member
Jul 28, 2021
115
45
28
I'm trying to find Connectx-4 dual port 25Gb MCX4121A-ACAT cards new, and failing.

I have found some 'Dell Mellanox CX4121C' server pulls, which appear to be the same card.

Will I be able to flash these Dell cards with the generic Mellanox firmware? Will they then function as a generic Mellanox card?
 

i386

Well-Known Member
Mar 18, 2016
2,987
953
113
33
Germany
Will I be able to flash these Dell cards with the generic Mellanox firmware? Will they then function as a generic Mellanox card?
I don't know about dell cards, I have hpe branded cx-4 that I flashed with the stock mellenax firmware. Some other user here in the forum have posted that they could flash their cx-4 (and cx-5) with the stock firmware, but then the cards would not work.

I would say get one card, try to get it working with the mellanox firmware and then make a post here in the forum what steps were necessary :)
 
  • Like
Reactions: ectoplasmosis

ectoplasmosis

Active Member
Jul 28, 2021
115
45
28
I don't know about dell cards, I have hpe branded cx-4 that I flashed with the stock mellenax firmware. Some other user here in the forum have posted that they could flash their cx-4 (and cx-5) with the stock firmware, but then the cards would not work.

I would say get one card, try to get it working with the mellanox firmware and then make a post here in the forum what steps were necessary :)

I wasn't brave enough and ordered a couple of allegedly genuine, new Mellanox cards from Ebay... Will see what turns up
 

Labs

Member
Mar 21, 2019
86
16
8
From what I experimented, if the card looks like the stock card it can be cross flashed. If the PCB or cooling part is different you must pay attention.
From my experiments the cards were not bricked but you cannot get the ports up and running so make sure you have a backup of the original firmware. Mellanox is selling the ASICs to many OEMs so they implement their own cards and they customize the firmware.
 
  • Like
Reactions: ectoplasmosis

xtantaudio

New Member
Apr 8, 2022
10
2
3
Looking for some assistance here. I have the ICX6650 flashed with the newest firmware and licenses as per the instructions here:

all the 10Gb ports appear to be working, the 2 10Gb breakout ports are confirmed 100% working, not sure yet on the 4 40Gb ports on the back, due to the following reason:

I also have 3 649281-B21 cards, 2 are V2 cards, one is a V4 card. The cards are flashed with the instructions posted at the beginning of this thread. Flashing went well, no issues. Using a windows server 2019 box to do the flashing.

the v2 cards never respond to me plugging in the QSFP cables. The V4 card does respond, but it lights up the connection and then turns off, sometimes multiple times per second. Windows shows no cable attached on the V2 Cards, even though cables are plugged in. I have 8 or so cables that I have tested with, 3 of which are from different manufactures. The cards show correctly in device manager and show as network adapters. I have tried both the VPI and the ETH settings with no luck.

I have also tried moving the cards from my desktop PC with windows server 2019 to my ESXi 7.03 boxes and to another desktop running windows 10, still no luck.

Any recommendations or advice ? I can provide screenshots, more information... anything anyone needs to be able to help.
 

xtantaudio

New Member
Apr 8, 2022
10
2
3
Do the cards work when doing a direct connection between two systems?
Yes, I just connected my Server 2019 desktop test PC to my ESXi 7.03 server (Lenovo RD640) together and they are showing as link up, 40Gb/s. I tried all 3 cards and they are all working directly connected together.
 

eduncan911

The New James Dean
Jul 27, 2015
537
380
63
eduncan911.com
Yes, I just connected my Server 2019 desktop test PC to my ESXi 7.03 server (Lenovo RD640) together and they are showing as link up, 40Gb/s. I tried all 3 cards and they are all working directly connected together.
Well, you've confirmed the cards are fine. :)

Hop over to the Brocade forum thread as they would know the switch-side of things. I know the ICX6610 doesn't clear the UPLINK state on a full reset. You have to perform a few extra steps to clear it.

Have you read through the ICX6650 setup guide from the Brocade God himself? ICX6650 - Fohdeesha Docs
 

xtantaudio

New Member
Apr 8, 2022
10
2
3
Well, you've confirmed the cards are fine. :)

Hop over to the Brocade forum thread as they would know the switch-side of things. I know the ICX6610 doesn't clear the UPLINK state on a full reset. You have to perform a few extra steps to clear it.

Have you read through the ICX6650 setup guide from the Brocade God himself? ICX6650 - Fohdeesha Docs
I was hoping it was not the switch :confused: but I was leaning in that direction. I followed the setup and licensing from the Brocade God :D His postis what led me to want to get the brocade switch and spend the 500 bucks.

I will head over there and see if anyone has any suggestions. I also sent a message to Fohdeesha to see if they could help at all.

Thank you for your response !!!! I appreciate the help. This has to be (IMO) the best forum group on the internet !

If anyone needs any Cisco ASA, switch or router stuff/help, hit me up, I want to return the favors here.
 
  • Like
Reactions: eduncan911

sadamahan

New Member
May 18, 2022
1
0
1
Great success with flashing firmware.

However, when the PC wakes from sleep, it loses its IP address. Have to stop and restart the network interface for network connectivity to be restored. Using default drivers in Windows 11.

Why is the card losing its IP address when PC wakes from sleep?

Ethernet adapter 10G Port 3:
Connection-specific DNS Suffix . : mydomain.com
Description . . . . . . . . . . . : Mellanox ConnectX-3 Ethernet Adapter #2
Physical Address. . . . . . . . . : 00-02-C9-45-67-D0
DHCP Enabled. . . . . . . . . . . : Yes
Autoconfiguration Enabled . . . . : Yes
Link-local IPv6 Address . . . . . : fe80::599f:7c08:53f6:e55a%2(Preferred)
Autoconfiguration IPv4 Address. . : 169.254.229.90(Preferred) <-- Seems to be picking an APIPA address
Subnet Mask . . . . . . . . . . . : 255.255.0.0

Default Gateway . . . . . . . . . :
DHCPv6 IAID . . . . . . . . . . . : 33555145
DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-27-5D-62-5F-24-4B-FE-DF-EF-35
DNS Servers . . . . . . . . . . . : 192.168.222.1
NetBIOS over Tcpip. . . . . . . . : Enabled

Drive Information:
Network adapter 10G Port 3 driver information:

Description . . . . . . . . . . : Mellanox ConnectX-3 Ethernet Adapter
Manufacturer . . . . . . . . . : Mellanox Technologies Ltd.
Provider . . . . . . . . . . . : Microsoft
Version . . . . . . . . . . . : 5.50.14695.0
Inf File Name . . . . . . . . . : C:\WINDOWS\INF\netmlx4eth63.inf
Inf File Date . . . . . . . . . : Saturday, June 5, 2021 12:04:45 PM
Section Name . . . . . . . . . : mlx4eth63_CX3.ndi
Hardware ID . . . . . . . . . . : mlx4\connectx-3_eth
Instance Status Flags . . . . . : 0x180200a
Device Manager Status Code . . : 0
IfType . . . . . . . . . . . . : 6
Physical Media Type . . . . . . : 14


Device Configuration:
Device #1:
----------

Device type: ConnectX3
Device: mt4099_pciconf0

Configurations: Next Boot
SRIOV_EN True(1)
NUM_OF_VFS 16
LINK_TYPE_P1 ETH(2)
LINK_TYPE_P2 ETH(2)
LOG_BAR_SIZE 3
BOOT_PKEY_P1 0
BOOT_PKEY_P2 0
BOOT_OPTION_ROM_EN_P1 False(0)
BOOT_VLAN_EN_P1 False(0)
BOOT_RETRY_CNT_P1 0
LEGACY_BOOT_PROTOCOL_P1 None(0)
BOOT_VLAN_P1 1
BOOT_OPTION_ROM_EN_P2 False(0)
BOOT_VLAN_EN_P2 False(0)
BOOT_RETRY_CNT_P2 0
LEGACY_BOOT_PROTOCOL_P2 None(0)
BOOT_VLAN_P2 1
IP_VER_P1 IPv4(0)
IP_VER_P2 IPv4(0)
CQ_TIMESTAMP True(1)

Do not have this problem on a different card that's Rev: A5 and has NUM_OF_VFS 8. Card with the problem is Rev: A2 and has NUM_OF_VFS 16 as you can see above. Could NUM_OF_VFS cause the card to lose its IP address when PC wakes from sleep? Moving a working card to another PC causes the problem to appear on that PC. Anything in the BIOS control this behavior?
 
Last edited: