Dell Mellanox connectx-3 to Arista 7050, no link?

[alijnclarke]

I recently purchased this card:

M9NW6 Dell Mellanox 40Gb/s CX324A CONNECTX-3 Dual Port QSFP+ Network Card LowPro | eBay

With the intention of placing it into my Dell R720 and using it to connect to my Arista 7050 qsfp switch. The card came and I installed it into the server, iDrac can see the config (as you'd expect with a dell branded nic) but I can'get get a link to my switch.

I'm using fs.com optics (arista coded on the switch side and mellanox on the card side) and a MTP fibre cable between - an fs.com tech confirmed the three parts were compatible (but please do say if they're wrong).

I've set the port to 40g forced on the switch but still can't get any kind of link. I initially thought that the card may of been in an infiniband mode or something like that, but it looks like that card does actually only support ethernet. I've updated it to the latest firmware using Dell Openmanage enterprise.

Does any one have any suggestions? If the card is faulty i'd like to return it so don't have much more time left.

Thanks

 

[PithyChats]

I recently purchased this card:

M9NW6 Dell Mellanox 40Gb/s CX324A CONNECTX-3 Dual Port QSFP+ Network Card LowPro | eBay

With the intention of placing it into my Dell R720 and using it to connect to my Arista 7050 qsfp switch. The card came and I installed it into the server, iDrac can see the config (as you'd expect with a dell branded nic) but I can'get get a link to my switch.

I'm using fs.com optics (arista coded on the switch side and mellanox on the card side) and a MTP fibre cable between - an fs.com tech confirmed the three parts were compatible (but please do say if they're wrong).

I've set the port to 40g forced on the switch but still can't get any kind of link. I initially thought that the card may of been in an infiniband mode or something like that, but it looks like that card does actually only support ethernet. I've updated it to the latest firmware using Dell Openmanage enterprise.

Does any one have any suggestions? If the card is faulty i'd like to return it so don't have much more time left.

Thanks

Are you sure the card supports ethernet? Do you have a link? A cursory search didn't turn that up.

 

[i386]

Did you "no shutdown" the interface on the switch? I had to run this command on my switch to make the link work. (Using qsfp dac cables though)

 

[alijnclarke]

Are you sure the card supports ethernet? Do you have a link? A cursory search didn't turn that up.

I'm not sure no, but a post on /r/homelab seemed to suggest that and the iDrac interface shows the card listed, if it is an infiniband only card I can return it.. so just need to confirm that's the case.

This is what iDrac shows
Imgur: The magic of the Internet

Any thoughts?

 

[alijnclarke]

Did you "no shutdown" the interface on the switch? I had to run this command on my switch to make the link work. (Using qsfp dac cables though)

Yeah tried that a few times, including a 'shutdown' and 'no shutdown' after setting 40g force.

 

[i386]

Do you use windows? If yes can you show the port protocol of the mellanox card in the device manager?

 

[fohdeesha]

Did you ever try linking it up *before* updating the firmware via dell? I had a dell branded connectx3 behave exactly like this when it had the "wrong" firmware, just would not link up even though everything appeared fine. Ended up compiling my own latest OEM (removing dell branding) firmware BIN file with mlxburn and flashing that, started working perfectly. Let me know if you want a quick guide on how to do so, it's only 3 commands or so once you have MFT installed

 

[alijnclarke]

Did you ever try linking it up *before* updating the firmware via dell? I had a dell branded connectx3 behave exactly like this when it had the "wrong" firmware, just would not link up even though everything appeared fine. Ended up compiling my own latest OEM (removing dell branding) firmware BIN file with mlxburn and flashing that, started working perfectly. Let me know if you want a quick guide on how to do so, it's only 3 commands or so once you have MFT installed

Absolutely would like the guide if at all possible.

I can't remember if i tired it before the firmware update, it's very possible that I didn't to be honest

 

[alijnclarke]

Do you use windows? If yes can you show the port protocol of the mellanox card in the device manager?

It's currently in an esxi box, but I can pull the card and put it in my windows machine, I'll try that now

 

[fohdeesha]

EDIT!!!!!: DO NOT FOLLOW THIS, FOLLOW THE NEWER GUIDE TO DIRECTLY FLASH THE LATEST FIRMWARE: https://forums.servethehome.com/ind...net-dual-port-qsfp-adapter.20525/#post-198015

install MFT: http://www.mellanox.com/page/management_tools

grab this archive, it's the latest mlx source files available:

Normal cards: http://content.mellanox.com/firmware/ConnectX3-rel-2_40_5030.tgz
"Pro" variants: http://content.mellanox.com/firmware/ConnectX3Pro-rel-2_40_5030.tgz

#start the service:
mst start

#get the PCI dev name:
mst status

#use that pci dev path for the following commands
#backup the current firmware
flint -d /dev/mst/mt4103_pci_cr0 ri orig_firmware.bin

#backup the current hardware config registers
flint -d /dev/mst/mt4103_pci_cr0 dc orig_firmware.ini

#use mlxburn to compile a new firmware bin using your hardware registers
#you'll need the orig_firmware.ini you generated earlier and the .mlx from the zip
mlxburn -fw fw-ConnectX3-rel.mlx -conf orig_firmware.ini -wrimage newFW.bin

#verify the bin you generated is good
flint -i newFW.bin verify

#finally flash it
flint -d /dev/mst/mt4103_pci_cr0 -i newFW.bin -allow_psid_change burn

cold reboot the system, when it comes back up do mst start then mst status to make sure it's come back up properly. If it's a vpi card you might have to configure it for ethernet, not sure

#might also be of use to query the cards current config
mlxconfig -d /dev/mst/mt4103_pci_cr0 query

 

[i386]

-allow_psid_change

This paramter "changes" the dell card to a mellanox card (on firmware level, serial/product number will still be dell sn/pn).

 

[fohdeesha]

that parameter doesn't change anything itself, it's the OEM firmware he compiled in the previous steps that does that. That argument just allows flint to flash it to the differently branded card (different PSID from the source bin). Without it flint will just say "nope, mismatched PSID's"

edit: actually, it's probably not even needed. compiling using the original firmware.ini should retain the PSID and everything else

 

[Foray]

Have you checked to see if your MTP/MPO cable is the correct type for your application?

Understanding Polarity in MPO System

 

[alijnclarke]

Apologies, quick update on this thread..

I returned the card and got an intel unit which works perfectly. The card I'd initially ordered must of been the wrong one or faulty.