ebay dell mellanox dual 10gbe

gzorn

Member
Jan 10, 2017
76
14
8
Well, with some more tinkering, I was able to get SR-IOV working on the cross-flashed CX322. I'll probably hold on to this now. The only other problem that I found - the activity LED's seem to be screwed up by the cross-flash.

At some point, I'll post a report on some of the details of the cross-flash adventure.
Now to get my hands on a full height bracket.
 

abq

Active Member
May 23, 2015
624
167
43
Let me know if you find a good source for full height brackets. I could use a few for various mellanox dual port cards.
 
  • Like
Reactions: gzorn

trumee

Member
Jan 31, 2016
220
12
18
52
I dont know about the cards posted here, but the cheap Mellanox ConnectX EN(MT26448) gave me problems with LACP and vlans to a 10G switch. The host was running vanilla FreeBD-11.0. I replaced the Mellanox with a Solarflare SFC9020 to get around the problem. Perhaps, the driver has been improved in the later versions.
 

kiteboarder

Active Member
May 10, 2016
108
57
28
43
Well, with some more tinkering, I was able to get SR-IOV working on the cross-flashed CX322. I'll probably hold on to this now. The only other problem that I found - the activity LED's seem to be screwed up by the cross-flash.

At some point, I'll post a report on some of the details of the cross-flash adventure.
Now to get my hands on a full height bracket.
Definitely would appreciate a cliff's notes version. I will be attempting this soon myself. Thanks in advance!
 
  • Like
Reactions: gzorn

gzorn

Member
Jan 10, 2017
76
14
8
@kiteboarder My only advice is to make doubly sure that you will really benefit from the flash (get features that you don't have with current firmware). I'm not necessarily sure that I gained all that much flashing my card (possibly SR-IOV, but even that i'm not sure about).
Basically, I followed the instructions in: Flash OEM (IBM) Mellanox ConnectX-3 EN with stock firmware?
I did bridge the 'flash' holes on the card with a piece of copper wire.
Then I used mlxfwburn (I think) to update the firmware to the very latest (it also wiped the old UEFI firmware out).
It mostly 'just worked', but there are little weird behaviors (no SFP module reading, LED weirdness, flash may not be write protected - even with jumper not bridged, etc. No guarantees, YMMV.

For SR-IOV under linux (proxmox v5beta), I think the Mellanox drivers get loaded in the init ramdisk, so one can either 1. Change the initrd to include the appropriate options to force the SR-IOV virtual functions or 2. run: modprobe -r mlx4_en ; modprobe mlx4_en after booting (assuming that you've specified your SR-IOV options in /etc/modprobe.d/
At one time, I thought I got the BIOS on the card to force the SR-IOV settings automagically, but I cannot reproduce that now. I've just changed too many things on the test server.

Hope this helps.
 
  • Like
Reactions: kiteboarder