Connectx-3 HP to Mellanox firmware help

[Jeff Robertson]

I've got my second set of ebay Mellanox cards and this time they are HP branded Connectx-3 cards:

649281-B21 656089-001 661685-001 HP IB FDR/EN 10/40GB 2P 544QSFP ADAPTER | eBay

I would like to get the firmware updated to the latest Mellanox release but am having no luck. I'm not sure which driver to download nor how to make it burn with different PSIDs. Any thoughts?

Here is the output from mst/flint:

C:\WINDOWS\system32>mst status
MST devices:
------------
mt4099_pci_cr0
mt4099_pciconf0

C:\WINDOWS\system32>flint -d mt4099_pci_cr0 query
Image type: FS2
FW Version: 2.30.3200
Rom Info: type=PXE version=3.4.146 devid=4099 proto=VPI
devid=4099 version_id=8025 type=CLP
Device ID: 4099
Description: Node Port1 Port2 Sys image
GUIDs: 80c16effff094830 80c16effff094831 80c16effff094832 80c16effff094833
MACs: 80c16e094831 80c16e094832
VSD:
PSID: HP_0280210019

 

[Patrick]

Whoa... that was some deal.

 

[whitey]

Details of how to query and write out PSID of current card and reflash w/ proper FW matrix is in my other thread to you a few days back where we updated your other card/s successfully. Imagine it is a VERY similar process. Send one to me and I'll glady flash/brick...I hold no liability haha j/k

 

[Jeff Robertson]

Haha, I'll ship both of them right over! I'm not positive as to which card this is so I don't know which file to download (or maybe they are all the same?) nor which .ini to edit once I do. The first time I downloaded a file it was an empty .zip file so that threw me for a bit. Here is the page that hosts the .ini firmware files, any thoughts on how to determine which one I need?

http://www.mellanox.com/page/custom_firmware_table

Thanks!

 

[Jeff Robertson]

If I download the file ConnectX3-rel-2_40_5000.tgz I get a bunch of .ini files, I can go from there if I can figure out which one relates to my card... These are the .ini files that I can extract:



I have a feeling it's MCX354A-FCC_Ax.ini but I don't want to brick the darn thing.

 

[whitey]

Compare Device ID: 4099 to the ConnectX-3 FW compat matrix (may have to use interim FW matrix), don't own any of the 3 series of this card so I have limited experience. Should be identical process after than as the 2 series.

This may help.

http://www.mellanox.com/page/firmware_HCA_FW_identification

EDIT: Probably one of these right, I see a lot of devid 4099, select proper FW image for card layout/features. If it's a ConnectX-3 VPI this should be the right url.

http://www.mellanox.com/page/firmware_table_ConnectX3IB

FYI, looks like there is a newer FW 2.40.5000 located here:

http://www.mellanox.com/page/custom_firmware_table

 

[Jeff Robertson]

It does look like it is one of the last three drivers on this page:

http://www.mellanox.com/page/firmware_table_ConnectX3IB

No idea how to determine which one though! Once I do how do I flash the file since my card has an HP PSID? With the connectx-2 I edited a .ini file to match... Looks like those .ini files I posted above are for the pro version and if I download the non pro version the zip file contains a single file with no extension.

 

[whitey]

Dev ID 4099 is NOT a PRO series so I don't think it is this MCX354A-FCC_Ax.ini

If I were a betting man it'd be this one.

MCX354A-TCBT MT_1090110028
ConnectX-3 VPI adapter card; dual-port QSFP; FDR10 IB (40Gb/s) and 10GigE; PCIe3.0x8 8GT/s; RoHS R6

PCI DevID: 4099

 

[whitey]

It does look like it is one of the last three drivers on this page:

http://www.mellanox.com/page/firmware_table_ConnectX3IB

No idea how to determine which one though! Once I do how do I flash the file since my card has an HP PSID? With the connectx-2 I edited a .ini file to match... Looks like those .ini files I posted above are for the pro version and if I download the non pro version the zip file contains a single file with no extension.

TIP - flint -d /dev/mst/mt26428_pci_cr0 dc > MT_0DD0120009.ini (to capture current ini out to file)

Then all you gotta do is use proper .mlx file and .ini captured if you really don't wanna hunt it down.

2cents.

 

[Jeff Robertson]

Ha, I agree, I wonder what happens if I flash the wrong one... I can't find any .ini/.mlx files to edit that match up with the non pro! The non pro download just gives me a single file without an extension. I have a feeling using the pro driver would be a very bad idea.

 

[Jeff Robertson]

I should also mention I did manage to flash both cards to 2.36.???? from HPs website but it appears to be the newest they have available.

 

[Marsh]

I'll send you the flash instruction.

Sorry for the delay.

 

[whitey]

I see the .mlx and .ini files in the .tgz here: (this has .mlx/.ini for your card from the link below and my process covers using this end-to-end)

Firmware DISCLAIMER

Also a what looks to be just the FW file already pre-packaged in the bin.zip here: (can skip this if using my process below)

http://www.mellanox.com/downloads/f..._5000-MCX354A-TCB_Ax-FlexBoot-3.4.746.bin.zip

Same process we followed last time w/ something similar to the following should do the trick.

mlxburn -fw fw-ConnectX3-rel.mlx -conf MCX354A-TCB_Ax.ini -wrimage fw-ConnectX3-rel-2_40_5000-MCX354A-TCB_Ax.bin

then

mstflint -d 03:00.0 -i fw-ConnectX3-rel-2_40_5000-MCX354A-TCB_Ax.bin b

BOOM, done deal.

I'd almost bet the house on this working!

 

[whitey]

I should also mention I did manage to flash both cards to 2.36.???? from HPs website but it appears to be the newest they have available.

I guess my question to all this updating is 'are you missing any features or the hardware not behaving properly' w/ 2.36 from HP installed or do you just want official OEM/Mellanox drivers on the card and latest/greatest?

 

[Jeff Robertson]

I'll send you the flash instruction.

Sorry for the delay.

Thanks, got it!

 

[Jeff Robertson]

I guess my question to all this updating is 'are you missing any features or the hardware not behaving properly' w/ 2.36 from HP installed or do you just want official OEM/Mellanox drivers on the card and latest/greatest?

I'll give you two guesses, as long as you guess latest/greatest you will get it right . I figure I might as well go all the way if I'm going to do it at all.

 

[whitey]

Hey @Marsh, if you wouldn't mind please attach or cross-link existing site link to the flashing procedure you use. May be beneficial to others

I assume my process in post #13 of this thread should be fairly accurate though. Always good to have plenty of documented resources for prosperity's sake and to grow the lumbering giant that is our collective wealth of harnessed knowledge.

 

[Marsh]

My CrossFlash_CX3.bat file
=================================================
echo Cross Flash HP to Mellanox CX3 firmware
mlxburn -fw fw-ConnectX3-rel.mlx -conf MCX354A-FCB_A2-A5.ini -wrimage HP_mlnx_firmware.bin
call flint -i HP_mlnx_firmware.bin verify
call flint -i HP_mlnx_firmware.bin query full
call flint -d mt4099_pciconf0 -i HP_mlnx_firmware.bin -allow_psid_change burn
=================================================

http://www.mellanox.com/page/custom_firmware_table
Download ConnectX3-rel-2_36_5150 or ConnectX3-rel-2_40_5000 from Mellanox download site
Download and install WinMFT_x64_4_5_0_31.exe

At the time of my cross flash HP cards to Mellanox psid, ConnectX3-rel-2_36_5150 was the most current firmware. Then ConnectX3-rel-2_40_5000 version came out.
I did not have a chance to directly cross flash HP card to Mellanox psid using ConnectX3-rel-2_40_5000 version.

Edit: Added download link to Mellanox site

 

[Marsh]

One thing to add, I flashed 20 HP cards with the crossflash_cx3 bat file without issue.

 

[patchate]

I'm kind of going off topic here, but I was wondering if any of you are using/plan on using this card with a 40Gbe switch via QSFP+ DAC cables. I've been trying to find ones that are reasonably priced on eBay, but I have no idea what's supposed to work.