@kiteboarder My only advice is to make doubly sure that you will really benefit from the flash (get features that you don't have with current firmware). I'm not necessarily sure that I gained all that much flashing my card (possibly SR-IOV, but even that i'm not sure about).
Basically, I followed the instructions in:
Flash OEM (IBM) Mellanox ConnectX-3 EN with stock firmware?
I did bridge the 'flash' holes on the card with a piece of copper wire.
Then I used mlxfwburn (I think) to update the firmware to the very latest (it also wiped the old UEFI firmware out).
It mostly 'just worked', but there are little weird behaviors (no SFP module reading, LED weirdness, flash may not be write protected - even with jumper not bridged, etc. No guarantees, YMMV.
For SR-IOV under linux (proxmox v5beta), I think the Mellanox drivers get loaded in the init ramdisk, so one can either 1. Change the initrd to include the appropriate options to force the SR-IOV virtual functions or 2. run: modprobe -r mlx4_en ; modprobe mlx4_en after booting (assuming that you've specified your SR-IOV options in /etc/modprobe.d/
At one time, I thought I got the BIOS on the card to force the SR-IOV settings automagically, but I cannot reproduce that now. I've just changed too many things on the test server.
Hope this helps.