Flashing stock Mellanox firmware to OEM (EMC) ConnectX-3 IB/Ethernet dual-port QSFP adapter

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
V

Valdar

New Member
Nov 22, 2021
1
0
1
Ethernet ports show as "cable unplugged"

I am setting up a new peer to peer Network
Windows Server 2022 + Windows 11

2 cards MCX354A-FCBT
Cable Dell 05NP8R SAS Kabel 40Gb QSFP+ / QSFP+ 1m

Driver Version : 5.50.14740.0
Firmware Version : 2.42.5000
Port Number : 1
Bus Type : PCI-E 8.0 Gbps x8
Link Speed : ----
Part Number : MCX354A-FCBT
Network Status : Disconnected
Adapter Friendly Name : Ethernet 4
IPv4 Address : 20.20.20.20
Cabling the card do the same card in another Windows box and setting both to Ethernet mode brings up the connection immediately.

any idea?
 
I

inf3rno

Member
Mar 24, 2021
36
1
8
What kind of transceivers do you use for 56Gbps? I found on the Mellanox website that only Mellanox MC2210411-SR4 is compatible with them and it is rare and more expensive than the HP cards I bought. Is it possible that cheap Cisco SR4 transceivers are compatible with these cards?
 
N

NablaSquaredG

Bringing 100G switches to homelabs
Aug 17, 2020
1,591
1,051
113
They are compatible with Mellanox devices (SX6036, CX354A)

Only Mellanox devices support 56G

Those transceivers require MTP / MPO cables... That's the only downside
 
I

inf3rno

Member
Mar 24, 2021
36
1
8
N

NablaSquaredG

Bringing 100G switches to homelabs
Aug 17, 2020
1,591
1,051
113
inf3rno said:
What kind of patch cable can I use for MTP / MPO? I wanted OM4 LC/LC, but I guess that is a completely different story.
Click to expand...
www.fs.com

MTP Trunkkabel 1m, 12F MTP Buchse OM4 Kabel, Typ B, Magenta - FS.com Deutschland

Kaufen Sie MTP Trunkkabel, 12 Fasern MTP Buchse OM4 50/125 Multimode Glasfaserkabel, Typ B Elite Plenum Magenta beim MTP/MPO Kabel Hersteller mit bestem Preis.
www.fs.com www.fs.com

This one for example

They are pretty costly though... And MTP is complex (Male / Female and different polarities). If you only have a single patch cable, no problem, but once you start with connectors and patch panels you really need to do some research
 
I

inf3rno

Member
Mar 24, 2021
36
1
8
NablaSquaredG said:
www.fs.com

MTP Trunkkabel 1m, 12F MTP Buchse OM4 Kabel, Typ B, Magenta - FS.com Deutschland

Kaufen Sie MTP Trunkkabel, 12 Fasern MTP Buchse OM4 50/125 Multimode Glasfaserkabel, Typ B Elite Plenum Magenta beim MTP/MPO Kabel Hersteller mit bestem Preis.
www.fs.com www.fs.com

This one for example

They are pretty costly though... And MTP is complex (Male / Female and different polarities). If you only have a single patch cable, no problem, but once you start with connectors and patch panels you really need to do some research
Click to expand...
I wanted to use OM4 LC/LC keystone modules to go through wall with it and have a proper keystone wall plate. I'll have only 4 devices and only one of them needs such a wall plate. I can keep the others near to each other. Looks like something like this exists, but I am not sure I can buy it from my country. Still good news. Adapter MPO-MPO, 10GB OM3, Aqua, Keystone Footprint Thanks for the help!
 
N

NablaSquaredG

Bringing 100G switches to homelabs
Aug 17, 2020
1,591
1,051
113
You could use a breakout to split the MTP cable into multiple LC connectors...

MTP is expensive. You should really reconsider whether it's worth the effort. It might be more sensible to go directly to 100G (Single Port CX-4 are relatively inexpensive, 100G-CWDM4 work with OS2 LC and are also relatively cheap)
 
I

inf3rno

Member
Mar 24, 2021
36
1
8
NablaSquaredG said:
You could use a breakout to split the MTP cable into multiple LC connectors...

MTP is expensive. You should really reconsider whether it's worth the effort. It might be more sensible to go directly to 100G (Single Port CX-4 are relatively inexpensive, 100G-CWDM4 work with OS2 LC and are also relatively cheap)
Click to expand...
Well I started with 10 Gbps. I want to finish this 56 Gbps project too. Maybe after it I'll check 100 Gbps, but for now this would be enough for M2 SSD speed. :) I'll check these MTP cables. I don't need really long ones 5-10 meters at most.
 
I

inf3rno

Member
Mar 24, 2021
36
1
8
NablaSquaredG said:
You could use a breakout to split the MTP cable into multiple LC connectors...

MTP is expensive. You should really reconsider whether it's worth the effort. It might be more sensible to go directly to 100G (Single Port CX-4 are relatively inexpensive, 100G-CWDM4 work with OS2 LC and are also relatively cheap)
Click to expand...
I checked meanwhile on amazon and ebay. You were right MTP is very expensive compared to OM4+LC and a lot more complicated as well. Since optical cables break easier than transceivers I'd rather go with the more expensive transceiver.
 
W

Wolfcastle

Member
Jan 3, 2022
55
30
18
Apologies if this has been asked before but is there a way to see transceiver information Windows?

Code: 
c:\Program Files\Mellanox\WinMFT>mlxlink -d mt4099_pci_cr0 -p 1

-E- Device is not supported
 
Last edited:
tinfoil3d

tinfoil3d

QSFP28
May 11, 2020
901
426
63
Japan
It's not strictly on topic but i recently got these dell-branded cx354a guys and easily reflashed them with stock.
What i found is though, repasting these NICs doesnt really help to keep them cooler. Maybe like 1-2 degrees. This particular card is reporting around 70C when no transceivers, and ramps up to 82C idle when you put them in(83C with AOC and above that with standalone transceivers). But as soon as you introduce some airflow(in my case i put a 12x12 fan next to the motherboard on a test bench) you get 30C off easily. The only thing these cards really need is a little bit of air. It's not that hard even in lenovo TMM nodes. And yes, 354a fits 1l nodes from lenovo. Using i350 bracket with these still won't align with screwholes but the bracket itself will keep NIC from moving around too much when changing transceivers.
In general, due to the bigger size of qsfp transceivers they end up sitting closer to the MCU, conducting several extra celsius to it. With DACs it's not an issue at all as they're not generating any noticeable heat whatsoever
 
eduncan911

eduncan911

The New James Dean
Jul 27, 2015
648
507
93
eduncan911.com
I read one person flashed to a Rev 6.

I'm looking at some cards now that are Rev AA, dated 2015. Could I flash those?
 
Last edited:
I

inf3rno

Member
Mar 24, 2021
36
1
8
NablaSquaredG said:
They are compatible with Mellanox devices (SX6036, CX354A)

Only Mellanox devices support 56G

Those transceivers require MTP / MPO cables... That's the only downside
Click to expand...
I had some time to think about this. I managed to find a 2x8 bifurcation riser for my mobo so I can add both dual port HP Mellanox cards finally. I still did not buy the transceivers and cables. I just checked that both the Mellanox MC2207411-SR4L and the Finisar FTL414QB2C has MPO connector. I guess the difference that the Mellanox is low power and can guarantee only 30 meter distance. I am not sure about the Finisar. They are about the same price on ebay currently. As of the cabling I guess my only option if I want to support keystone modules is using 1x MPO -> 4x LC-LC breakout cables. As of the distance, 30m is enough currently, but later when I move to my new house I'll need around 100m distance for at least one of my computers, so I think the Finisar is a better choice. Am I right about this?

Meanwhile I checked, both the Finisar and the Mellanox are 12 core MPOs, so I guess I need 1x MPO -> 6x LC-LC breakout cables, at least it makes sense this way. I am still not sure about the directions you mentioned, or whether it matters in what order the LC-LC pairs I connect. For now I'll buy 2 DAC cables to test if everything works properly and I'll take care about the fiber optics later when the upper part is clear.
 
Last edited:
L

LodeRunner

Active Member
Apr 27, 2019
553
235
43
Oh I just grabbed the first Google hit to show that they do exist. That fiber you linked looks like it's priced that way because it's "custom" (all lengths they offer from 0.15m to 0.9m are the same price for example). I sure wouldn't pay that for those fiber lengths.
 
I

inf3rno

Member
Mar 24, 2021
36
1
8
LodeRunner said:
There are MPO/MPT keystones.
Click to expand...
Thanks! I did not know it. Is there anything else I should know about this MPO/MPT standard? Does the direction matter? How long is the shortest cable possible I could use between keystone modules? What is the minimum bending radius for such cables? Are there flexible ones?
 
I

inf3rno

Member
Mar 24, 2021
36
1
8
LodeRunner said:
Oh I just grabbed the first Google hit to show that they do exist. That fiber you linked looks like it's priced that way because it's "custom" (all lengths they offer from 0.15m to 0.9m are the same price for example). I sure wouldn't pay that for those fiber lengths.
Click to expand...
Afaik the connector part is very expensive by these cables, while the fibers are relative cheap, that's why length does not matter that much.
 
You must log in or register to reply here.
Top Bottom