Mellanox ConnectX-3 SFP+ Nic not being detected in TrueNAS/Supermicro X8DTN+

Swillyums

New Member
Feb 19, 2021
6
0
1
I recently upgraded my network to 10gb, but I can't get my TrueNAS rig to recognize it.

This is the ebay listing for the product I bought: https://www.ebay.ca/itm/2PCS-MCX341...3?pageci=80ea2655-f689-46eb-adcd-183105ef37f2

Both cards are tested and working in my PC (Windows) and my Dell R710 (ESXI).

My motherboard is a Supermicro X8DTN+.

I have tried the card in several PCIE slots, with and without a riser. It fails to show up in pciconf -lv or anywhere that I have checked. It lights up and flashes when I connect a DAC cable.

I would love to hear anyone's suggestions of things I could try. As far as I've seen, everyone says that it should just work. They card is supported with FreeNAS; the person I bought it to said that he's never had any issues with SFP+ cards in the Supermicro servers, etc.

I'm not really sure what to try, or what the problem may be.

Please forgive me as I would put my knowledge at beginner to intermediate.

Thanks so much for the help.
 

Bjorn Smith

Active Member
Sep 3, 2019
341
143
43
Have you tried using the card in the X8DTN+ with something else than FreeBSD?

It could be that it is that specific board that does not like the card?

I mean it is quite an old board with only pci-e 2.0 - and even though pci-e 3.0 is compatible with 2.0 it might be "flaky" on an old board like that.
 

Babbage

New Member
Feb 17, 2021
10
1
3
Hi there I just did this, please read the whole thread, but here is my post in that thread.

Get the Mellanox mxup utility, chmod +x it, run it with no arguments to see what cards you have installed.
Dual cpus? NUMA? put nics in CPU0 slots if you can.

Then: find the newest firmware for your model card, use the output of mlxup to determine exact model.

Download the firmware and run mlxup -d <folder> for it to update the cards. If you have (2) cards, put them both in and update both to the same current supported firmware.

Hands off no reboot or power down when you are flashing - or the cards become "bricks" good luck.
Also: old Motherboard? Make sure you have latest bios. Boot to ufei shell (if it has that?) and update bios from usb stick.
 

Swillyums

New Member
Feb 19, 2021
6
0
1
Have you tried using the card in the X8DTN+ with something else than FreeBSD?

It could be that it is that specific board that does not like the card?

I mean it is quite an old board with only pci-e 2.0 - and even though pci-e 3.0 is compatible with 2.0 it might be "flaky" on an old board like that.
I have not. I very much expect that it would not work if it isn't being detected at all in Freenas, but I could try it this weekend. If that's the case, I guess I need a whole new system?
 

Swillyums

New Member
Feb 19, 2021
6
0
1
Hi there I just did this, please read the whole thread, but here is my post in that thread.

Get the Mellanox mxup utility, chmod +x it, run it with no arguments to see what cards you have installed.
Dual cpus? NUMA? put nics in CPU0 slots if you can.

Then: find the newest firmware for your model card, use the output of mlxup to determine exact model.

Download the firmware and run mlxup -d <folder> for it to update the cards. If you have (2) cards, put them both in and update both to the same current supported firmware.

Hands off no reboot or power down when you are flashing - or the cards become "bricks" good luck.
Also: old Motherboard? Make sure you have latest bios. Boot to ufei shell (if it has that?) and update bios from usb stick.
Thanks for this. I did just that.

Unfortunately I am getting the following error: -E- PSID mismatch. The PSID on flash (MT_1270112023) differs from the PSID in the given image (MT_1270114023).

Turns out I need to use their custom firmware table to find the right one. Unfortunately they seem to have taken that table down, and I'm not totally sure where to look for the correct firmware.

Dual socket, but single CPU in CPU0 socket. I tried several slots, no luck.

I believe that the motherboard does not have UEFI shell. I should update the BIOS, though.
 

i386

Well-Known Member
Mar 18, 2016
2,319
640
113
31
Germany
Try mlxburn, it should be able to update oem cards too.

If that doesn't work you could try to crassflash the card with the friwmare you mentioned and the argument "allow_psid_change" (This could brick your card!)
 

Bjorn Smith

Active Member
Sep 3, 2019
341
143
43
Unfortunately I am getting the following error: -E- PSID mismatch. The PSID on flash (MT_1270112023) differs from the PSID in the given image (MT_1270114023).
At least that is partially good news - it means the card is being detected by the pci-e bus - so the server is probably not too old.

I have also updated "HP" branded mellanox cards to stock mellanox cards using "allow_psid_change" - and that gave no issues.

And you can always take a backup of the current firmware just in case the card ends up i worse.

Dual socket, but single CPU in CPU0 socket. I tried several slots, no luck.
Find the motherboard manual and see what slots are active for when you only have a CPU in slot0 - then you don't have to guess what pci-e slots are usable.
 

adarobin

New Member
Feb 21, 2021
1
0
1
I just ordered some similar cards off eBay that showed MCX341A-XCEN as the model in the pictures. I poked around a bit on the Mellanox site and it looks like if you stick under current versions but then pick version 2.31.1598 then MCX341A-XCEN is listed with PSN MT_1270112023.
 

Swillyums

New Member
Feb 19, 2021
6
0
1
At least that is partially good news - it means the card is being detected by the pci-e bus - so the server is probably not too old.

I have also updated "HP" branded mellanox cards to stock mellanox cards using "allow_psid_change" - and that gave no issues.

And you can always take a backup of the current firmware just in case the card ends up i worse.
Actually I was attempting to do the update on my computer, not my server. I have now successfully updated the card, but it seems that this was not the issue. It won't allow me to force it into ethernet mode. It may be that this model is always and only in ethernet mode, and has no infiniband. Other than the lights on the card flashing, there is no indication whatsoever that the motherboard is aware of the card.

Find the motherboard manual and see what slots are active for when you only have a CPU in slot0 - then you don't have to guess what pci-e slots are usable.
I swapped out the HBA card with the SFP+ card, so I know the slot is working. I know the other slot I tried it in is working as well.[/QUOTE][/QUOTE]
 
Last edited:

Swillyums

New Member
Feb 19, 2021
6
0
1
@Swillyums You've seen the reply on the FN forum?
Regarding the change in the way 12.0 handles loading the drivers? I did, but that isn't my problem. I downloaded windows to check, and it just is not working. It's absolutely the server recognizing the card, not the OS.
 

Rand__

Well-Known Member
Mar 6, 2014
5,162
1,070
113
ah ok, then i confused the boxes, sorry.

Have you tried the old taping trick? that was a thing with LSI cards and (i think) the 3.3v supply.

there are threads on here too, but didnt have the correct keywords apparently