Dell Mellanox connectx-3 to Arista 7050, no link?

alijnclarke

New Member
Feb 9, 2017
10
1
1
28
I recently purchased this card:

M9NW6 Dell Mellanox 40Gb/s CX324A CONNECTX-3 Dual Port QSFP+ Network Card LowPro | eBay

With the intention of placing it into my Dell R720 and using it to connect to my Arista 7050 qsfp switch. The card came and I installed it into the server, iDrac can see the config (as you'd expect with a dell branded nic) but I can'get get a link to my switch.

I'm using fs.com optics (arista coded on the switch side and mellanox on the card side) and a MTP fibre cable between - an fs.com tech confirmed the three parts were compatible (but please do say if they're wrong).

I've set the port to 40g forced on the switch but still can't get any kind of link. I initially thought that the card may of been in an infiniband mode or something like that, but it looks like that card does actually only support ethernet. I've updated it to the latest firmware using Dell Openmanage enterprise.

Does any one have any suggestions? If the card is faulty i'd like to return it so don't have much more time left.

Thanks :)
 

PithyChats

Active Member
Feb 3, 2015
150
68
28
I recently purchased this card:

M9NW6 Dell Mellanox 40Gb/s CX324A CONNECTX-3 Dual Port QSFP+ Network Card LowPro | eBay

With the intention of placing it into my Dell R720 and using it to connect to my Arista 7050 qsfp switch. The card came and I installed it into the server, iDrac can see the config (as you'd expect with a dell branded nic) but I can'get get a link to my switch.

I'm using fs.com optics (arista coded on the switch side and mellanox on the card side) and a MTP fibre cable between - an fs.com tech confirmed the three parts were compatible (but please do say if they're wrong).

I've set the port to 40g forced on the switch but still can't get any kind of link. I initially thought that the card may of been in an infiniband mode or something like that, but it looks like that card does actually only support ethernet. I've updated it to the latest firmware using Dell Openmanage enterprise.

Does any one have any suggestions? If the card is faulty i'd like to return it so don't have much more time left.

Thanks :)

Are you sure the card supports ethernet? Do you have a link? A cursory search didn't turn that up.
 

i386

Well-Known Member
Mar 18, 2016
2,658
774
113
32
Germany
Did you "no shutdown" the interface on the switch? I had to run this command on my switch to make the link work. (Using qsfp dac cables though)
 

alijnclarke

New Member
Feb 9, 2017
10
1
1
28
Did you "no shutdown" the interface on the switch? I had to run this command on my switch to make the link work. (Using qsfp dac cables though)
Yeah tried that a few times, including a 'shutdown' and 'no shutdown' after setting 40g force.
 

i386

Well-Known Member
Mar 18, 2016
2,658
774
113
32
Germany
Do you use windows? If yes can you show the port protocol of the mellanox card in the device manager?
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,277
2,358
113
30
fohdeesha.com
Did you ever try linking it up *before* updating the firmware via dell? I had a dell branded connectx3 behave exactly like this when it had the "wrong" firmware, just would not link up even though everything appeared fine. Ended up compiling my own latest OEM (removing dell branding) firmware BIN file with mlxburn and flashing that, started working perfectly. Let me know if you want a quick guide on how to do so, it's only 3 commands or so once you have MFT installed
 

alijnclarke

New Member
Feb 9, 2017
10
1
1
28
Did you ever try linking it up *before* updating the firmware via dell? I had a dell branded connectx3 behave exactly like this when it had the "wrong" firmware, just would not link up even though everything appeared fine. Ended up compiling my own latest OEM (removing dell branding) firmware BIN file with mlxburn and flashing that, started working perfectly. Let me know if you want a quick guide on how to do so, it's only 3 commands or so once you have MFT installed
Absolutely would like the guide if at all possible.

I can't remember if i tired it before the firmware update, it's very possible that I didn't to be honest
 

alijnclarke

New Member
Feb 9, 2017
10
1
1
28
Do you use windows? If yes can you show the port protocol of the mellanox card in the device manager?
It's currently in an esxi box, but I can pull the card and put it in my windows machine, I'll try that now
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,277
2,358
113
30
fohdeesha.com
EDIT!!!!!: DO NOT FOLLOW THIS, FOLLOW THE NEWER GUIDE TO DIRECTLY FLASH THE LATEST FIRMWARE: https://forums.servethehome.com/ind...net-dual-port-qsfp-adapter.20525/#post-198015

install MFT: http://www.mellanox.com/page/management_tools

grab this archive, it's the latest mlx source files available:

Normal cards: http://content.mellanox.com/firmware/ConnectX3-rel-2_40_5030.tgz
"Pro" variants: http://content.mellanox.com/firmware/ConnectX3Pro-rel-2_40_5030.tgz

#start the service:
mst start

#get the PCI dev name:
mst status

#use that pci dev path for the following commands
#backup the current firmware
flint -d /dev/mst/mt4103_pci_cr0 ri orig_firmware.bin

#backup the current hardware config registers
flint -d /dev/mst/mt4103_pci_cr0 dc orig_firmware.ini

#use mlxburn to compile a new firmware bin using your hardware registers
#you'll need the orig_firmware.ini you generated earlier and the .mlx from the zip
mlxburn -fw fw-ConnectX3-rel.mlx -conf orig_firmware.ini -wrimage newFW.bin

#verify the bin you generated is good
flint -i newFW.bin verify

#finally flash it
flint -d /dev/mst/mt4103_pci_cr0 -i newFW.bin -allow_psid_change burn

cold reboot the system, when it comes back up do mst start then mst status to make sure it's come back up properly. If it's a vpi card you might have to configure it for ethernet, not sure

#might also be of use to query the cards current config
mlxconfig -d /dev/mst/mt4103_pci_cr0 query
 
Last edited:

fohdeesha

Kaini Industries
Nov 20, 2016
2,277
2,358
113
30
fohdeesha.com
that parameter doesn't change anything itself, it's the OEM firmware he compiled in the previous steps that does that. That argument just allows flint to flash it to the differently branded card (different PSID from the source bin). Without it flint will just say "nope, mismatched PSID's"

edit: actually, it's probably not even needed. compiling using the original firmware.ini should retain the PSID and everything else
 

alijnclarke

New Member
Feb 9, 2017
10
1
1
28
Apologies, quick update on this thread..

I returned the card and got an intel unit which works perfectly. The card I'd initially ordered must of been the wrong one or faulty.