Flashing stock Mellanox firmware to OEM (EMC) ConnectX-3 IB/Ethernet dual-port QSFP adapter

Hindsight

Member
Mar 28, 2016
55
13
8
38
Are you on Windows 10? i have a CX-3 in mine now. I actually used my Win10 to crossflash all 3 CX-3 cards. The weirdest issue I had was after each boot of the first card the computer would green screen. I'd let it finish then power it off and back on and it booted fine after that. I used it to flash my 6012 in IB too. You can download the windows drivers on their website.

But you can't get your mobo to post at all? Try a different PCI-E slot?
 

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
656
233
43
Yeah, it's not an OS boot issue, the motherboard is hanging during POST. I haven't had a chance to try again after updating firmware so I'll do that today when I get home.

I've tried two of the 16x pcie slots so far (slots 2 and 3) but I'll give slot 1 a go if I can't get it working otherwise.

My other machine boots with either card installed so there's no problem with the adapters themselves. I just hope I'm not dealing with some janky Asus bios issue here.
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,959
1,783
113
29
fohdeesha.com
does your BIOS have any setting related to "above 4G decoding" or "LOG BAR space"? if so, try switching them on then see if it boots

and yes of course update the boards BIOS if it's not totally up to date
 

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
656
233
43
BIOS is current, just tested a freshly flashed adapter in every PCIe slot in the board. Above 4G decoding is switched on in BIOS.

When I feel like going under my desk again to swap cards I'll try setting LOG_BAR_SIZE = 0 on one of the boards and I'll try again, for the moment I'm just going to roll with the spare CX-2 I have kicking around.
 

Hindsight

Member
Mar 28, 2016
55
13
8
38
So I do see
Burning FS2 image failed: Device/Image mismatch: FW image file cannot be programmed to device ConnectX-3 A1, it is intended for: ConnectX-3Pro only
And I see where that is in Mellanox/mstflint

Wondering if I can just fork the distro and remove that check.


return errmsg(MLXFW_DEVICE_IMAGE_MISMATCH_ERR, "FW image file cannot be programmed to device %s, it is intended for: %s only",
curr_hw_id_name, supp_hw_id_list);
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,959
1,783
113
29
fohdeesha.com
That's one way of doing it, another is just using the low level flash sector access commands in flint to dump it straight to flash bypassing their check tool. There's also some arguments designed for OEM use that bypass most checks, like "BurnBlock", I've been meaning to test a few of these methods out but haven't had a chance yet. Might do it tonight

EDIT: note that the ConnectX-3 A1 revisions do not support the faster 427mhz clock as far as I remember, so if you did flash that it would most likely stop working. I believe only the a2 and up revisions support the 427mhz clock which enables the 40gbE ethernet and x3 Pro firmware
 

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
656
233
43
So I do see And I see where that is in Mellanox/mstflint

Wondering if I can just fork the distro and remove that check.
That's exactly what I was planning to do. I think one of us is going to have to just take the plunge and see what happens.

HP branded QCBT adapters are still $35 on eBay so it's not an extreme risk.
 

Hindsight

Member
Mar 28, 2016
55
13
8
38
I went ahead and did the burnblock, didn't know that existed. So I couldn't query it anymore. I rebooted and it doesn't show up anymore. Prob dont have any time tonight to see if I can bring it back but yeah it was a cheap card and figured I'd risk it.

Code:
C:\Users\j\Downloads\40gb\fw-ConnectX3Pro-rel-2_42_5000-MCX354A-FCC_Ax-FlexBoot-3.4.752.bin>flint -d mt4099_pci_cr0 -i fw-ConnectX3Pro-rel-2_42_5000-MCX354A-FCC_Ax-FlexBoot-3.4.752.bin  bb

Block burn: The given image will be burnt as is. No fields (such
as GUIDS,VSD) are taken from current image on flash.
Burn process will not be failsafe. No checks will be performed.
ALL flash, including the Invariant Sector will be overwritten.
If this process fails, computer may remain in an inoperable state.

 Do you want to continue ? (y/n) [n] : y
OK


C:\Users\j\Downloads\40gb\fw-ConnectX3Pro-rel-2_42_5000-MCX354A-FCC_Ax-FlexBoot-3.4.752.bin>mlxburn -query -d mt4099_pci_cr0
-E- flint query failed : -E- Failed to query Device: mt4099_pci_cr0.(null)
child process exited abnormally

C:\Users\j\Downloads\40gb\fw-ConnectX3Pro-rel-2_42_5000-MCX354A-FCC_Ax-FlexBoot-3.4.752.bin>mlxburn -query -d mt4103_pci_cr0
-E- Can not auto detect device type: Can't access device mt4103_pci_cr0: mopen: No such device. Please check the given device.

C:\Users\j\Downloads\40gb\fw-ConnectX3Pro-rel-2_42_5000-MCX354A-FCC_Ax-FlexBoot-3.4.752.bin>
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,959
1,783
113
29
fohdeesha.com
see the edit I made in my post 10 minutes or so ago, it's most likely because you had an A1 revision card. the flash can recovered/written to using the i2c headers if you have something like a raspberry pi laying around. FYI anyone reading, please do not try my suggestion above until I post thorough instructions, you also need to backup the VSD/MACs/GUIDs of your card before using burnblock as it wipes all of these
 

Hindsight

Member
Mar 28, 2016
55
13
8
38
Yeah I have a few laying around. Not sure when I'll get to it though. is there a guide to reflash via ic2?
 

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
656
233
43
see the edit I made in my post 10 minutes or so ago, it's most likely because you had an A1 revision card. the flash can recovered/written to using the i2c headers if you have something like a raspberry pi laying around. FYI anyone reading, please do not try my suggestion above until I post thorough instructions, you also need to backup the VSD/MACs/GUIDs of your card before using burnblock as it wipes all of these
I think both of the OEM HP cards I picked up on eBay are rev A2, can anyone confirm?

https://photos.app.goo.gl/2RfkPDFHFBWQrQFk7

Don't worry, I'm not jumping into flashing tonight.
 

Hindsight

Member
Mar 28, 2016
55
13
8
38
Believe it told me I was Rev 1 when I did the normal flash and it denied me to to HW ID

Also what is the HP model number?
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,959
1,783
113
29
fohdeesha.com
I think both of the OEM HP cards I picked up on eBay are rev A2, can anyone confirm?

https://photos.app.goo.gl/2RfkPDFHFBWQrQFk7

Don't worry, I'm not jumping into flashing tonight.
The version on the larger sticker (that also has SKU) indicates the card revision - However I am now having my doubts about some things - My card is also labelled as an A2 on the back, and this holds up as I was able to flash the 40gbE ethernet firmware onto it, which ups the clock to 427MHZ (same as the Pro image), and that works fine. running "mlxfwmanager --query" also tells me "MCX354A-FCB_A2-A5"

However just for a test I tried flashing using the normal burn command, and got the same message indicating it's actually an A1 - "FW image file cannot be programmed to device ConnectX-3 A1, it is intended for: ConnectX-3Pro only"

I'm going to do a little more digging and look at the source board config files for the Pro firmware to see if it's perhaps running at an even higher clock, then I'll continue flashing using the force/flash commands. This particular CX3 has broken QSFP cages so it's retired anyway, so I'm not terribly bothered about bricking it
 

Hindsight

Member
Mar 28, 2016
55
13
8
38
So you said that has the 40GB but I crossflashed QCBT to the FCBT and I see 40GB support with it on my Rev 1 cards. I should be able to get 56 GB too with FDR canles however I don't have PCIe3 so I won't be able to test how much I can push out of this one.
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,959
1,783
113
29
fohdeesha.com
Are you saying you successfully flashed FCBT firmware (40gb ethernet support, not just 40gb infiniband) to a card that has an "rev: A1" sticker on the back?
 

Hindsight

Member
Mar 28, 2016
55
13
8
38
Are you saying you succesfully flashed FCBT firmware (40gb ethernet support, not just 40gb infiniband) to a card that has an "rev: A1" sticker on the back?
Correct, there are a few of us who have been doing this. Buy the cheaper QCBT and flash FCBT. It has 56gb ethernet support which goes along with the converted emc 6012 switch

Imgur
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,959
1,783
113
29
fohdeesha.com
Sure, QCBT > FCBT is definitely supported, however Mellanox's notes say it will only work on Rev A2 and up, they state the Rev A1 ASIC will not run properly at the increased 427MHZ clock rate - the firmware image is even named "2_42_5000-MCX354A-FCB_A2-A5" as in A2 - A5 rev boards only. That's quite interesting that it works, I wonder if MLNX was just being cautious with their recommendations
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,959
1,783
113
29
fohdeesha.com
Actually it seems the above is correct, A1 cards will not support it, but you have a much newer card, after they changed revision schema from A-x - it seems once they hit revision A5, the next revision up from that started at decimal 01 (no leading A) which is what you have - hence the date of 2017 - my A4 revision cards are as old as 2014
 

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
656
233
43
Believe it told me I was Rev 1 when I did the normal flash and it denied me to to HW ID

Also what is the HP model number?
HP Model is 649281-B21, it's an HP branded FCBT, $35 source is here:
HP 649281-B21 Infiniband 10/40GB Dual Port Adapter Low Profile Bracket | eBay

This is what the board shipped with:
Code:
Querying Mellanox devices firmware ...
 
 Device #1:
 ----------
 
   Device Type:      ConnectX3
   Part Number:      649281-B21_Bx
   Description:      HP IB 4X FDR CX-3 PCI-e G3 Dual Port HCA
   PSID:             HP_0280210019
   PCI Device Name:  /dev/mst/mt4099_pci_cr0
   Port1 GUID:       --
   Port2 GUID:       --
   Versions:         Current        Available
      FW             2.11.0310      N/A      
      PXE            3.3.0900       N/A      
 
   Status:           No matching image found
And here it is flashed to FCBT fw:
Code:
[root@meta mlx_firmware]# mlxfwmanager --query
Querying Mellanox devices firmware ...

Device #1:
----------

  Device Type:      ConnectX3
  Part Number:      MCX354A-FCB_A2-A5
  Description:      ConnectX-3 VPI adapter card; dual-port QSFP; FDR IB (56Gb/s) and 40GigE; PCIe3.0 x8 8GT/s; RoHS R6
  PSID:             MT_1090120019
  PCI Device Name:  /dev/mst/mt4099_pci_cr0
  Port1 MAC:        --
  Port2 GUID:       --
  Versions:         Current        Available
     FW             2.42.5000      2.42.5000
     PXE            3.4.0752       3.4.0752 

  Status:           Up to date
This isn't the same board in the photo above, but afaik they're both A2 revision.

edit: I get the same message referencing an A1 revision board when attempting to flash the C-X3 Pro fw:
Code:
[root@meta mlx_firmware]# !2056
flint -d /dev/mst/mt4099_pci_cr0 -i fw-ConnectX3Pro-rel-2_42_5000-MCX354A-FCC_Ax-FlexBoot-3.4.752.bin -allow_psid_change burn

   Current FW version on flash:  2.42.5000
   New FW version:               2.42.5000

   Note: The new FW version is the same as the current FW version on flash.

 Do you want to continue ? (y/n) [n] : y


   You are about to replace current PSID on flash - "MT_1090120019" with a different PSID - "MT_1090111019".
   Note: It is highly recommended not to change the PSID.

 Do you want to continue ? (y/n) [n] : y
-E- Burning FS2 image failed: Device/Image mismatch: FW image file cannot be programmed to device ConnectX-3 A1, it is intended for: ConnectX-3Pro only
 
Last edited: