Mellanox cards that can support 40gb

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
J

Jamy

Member
Mar 29, 2017
44
6
8
48
Hi all,

Can someone point me to the Mellanox card models that can be flashed to 40GB? I need 3 of them for my XCP-NG and TrueNAS servers. I searched, but I'm having an issue finding model numbers.

Thanks,
 
M

mach3.2

Active Member
Feb 7, 2022
132
87
28
MCX353/MCX354 (single/dual port)

Flash the FCBT firmware if you order the QCBT variant.
 
J

Jamy

Member
Mar 29, 2017
44
6
8
48
What about the TCBT variant? Also, if I buy the FCBT I assume that means I would only need to flash with the latest firmware?

Thanks
 
tinfoil3d

tinfoil3d

QSFP28
May 11, 2020
880
404
63
Japan
Yes

I've a bunch of various OEM-labeled 354 cards and flashed all of them with the 40G firmware no problem. I mostly had HP, cisco and QCBT labeled NICs, can't really check them all but I have a lot of them running.
Just a side note, you're looking at max theoretical bandwidth of 64gbps (8x 8gbps pcie3 lanes). You won't get 80 from both ports.
 
J

Jamy

Member
Mar 29, 2017
44
6
8
48
tinfoil3d said:
Yes

I've a bunch of various OEM-labeled 354 cards and flashed all of them with the 40G firmware no problem. I mostly had HP, cisco and QCBT labeled NICs, can't really check them all but I have a lot of them running.
Just a side note, you're looking at max theoretical bandwidth of 64gbps (8x 8gbps pcie3 lanes). You won't get 80 from both ports.
Click to expand...
So I updated the firmware and I cannot get carrier light from the card. I confirmed port 1 is in ethernet mode. I am using AOI QSFP+ modules and trying to connect to my 10gb Ubiquiti switch. The manual says it should be able to run at 10gb. The OS (XCP-NG) shows the card as recognized. Could this be incompatible QSFP+ modules?
 
R

Rttg

Member
May 21, 2020
71
47
18
Jamy said:
Could this be incompatible QSFP+ modules?
Click to expand...
It’s more likely they’re stuck in VPI/IB mode (I’ve had to manually switch mine over to ethernet mode). I haven’t had any transceiver compatibility issues with the Mellanox card itself, but it’s also possible that your Ubiquiti switch is picky
 
J

Jamy

Member
Mar 29, 2017
44
6
8
48
Rttg said:
It’s more likely they’re stuck in VPI/IB mode (I’ve had to manually switch mine over to ethernet mode). I haven’t had any transceiver compatibility issues with the Mellanox card itself, but it’s also possible that your Ubiquiti switch is picky
Click to expand...
Mlxconfig shows both ports in ETH mode.


Code: 
Device #1:
----------

Device type:    ConnectX3
Device:         /dev/mst/mt4099_pci_cr0

Configurations:                                      Next Boot
        SRIOV_EN                                    False(0)
        NUM_OF_VFS                                  8
        LINK_TYPE_P1                                ETH(2)
        LINK_TYPE_P2                                ETH(2)
        LOG_BAR_SIZE                                3
        BOOT_PKEY_P1                                0
        BOOT_PKEY_P2                                0
        BOOT_OPTION_ROM_EN_P1                       False(0)
        BOOT_VLAN_EN_P1                             False(0)
        BOOT_RETRY_CNT_P1                           0
        LEGACY_BOOT_PROTOCOL_P1                     None(0)
        BOOT_VLAN_P1                                1
        BOOT_OPTION_ROM_EN_P2                       False(0)
        BOOT_VLAN_EN_P2                             False(0)
        BOOT_RETRY_CNT_P2                           0
        LEGACY_BOOT_PROTOCOL_P2                     None(0)
        BOOT_VLAN_P2                                1
        IP_VER_P1                                   IPv4(0)
        IP_VER_P2                                   IPv4(0)
        CQ_TIMESTAMP                                True(1)

I've tried the transceivers and cables on another NIC and they are both good. I however cannot get any carrier light on the Mellanox card. Do I have a bad card? Anything else I should try?
 
Last edited:
You must log in or register to reply here.
Top Bottom