Mellanox cards that can support 40gb

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Jamy

Member
Mar 29, 2017
44
6
8
48
Hi all,

Can someone point me to the Mellanox card models that can be flashed to 40GB? I need 3 of them for my XCP-NG and TrueNAS servers. I searched, but I'm having an issue finding model numbers.

Thanks,
 

mach3.2

Active Member
Feb 7, 2022
124
80
28
MCX353/MCX354 (single/dual port)

Flash the FCBT firmware if you order the QCBT variant.
 

Jamy

Member
Mar 29, 2017
44
6
8
48
What about the TCBT variant? Also, if I buy the FCBT I assume that means I would only need to flash with the latest firmware?

Thanks
 

tinfoil3d

QSFP28
May 11, 2020
860
394
63
Japan
Yes

I've a bunch of various OEM-labeled 354 cards and flashed all of them with the 40G firmware no problem. I mostly had HP, cisco and QCBT labeled NICs, can't really check them all but I have a lot of them running.
Just a side note, you're looking at max theoretical bandwidth of 64gbps (8x 8gbps pcie3 lanes). You won't get 80 from both ports.
 

Jamy

Member
Mar 29, 2017
44
6
8
48
Yes

I've a bunch of various OEM-labeled 354 cards and flashed all of them with the 40G firmware no problem. I mostly had HP, cisco and QCBT labeled NICs, can't really check them all but I have a lot of them running.
Just a side note, you're looking at max theoretical bandwidth of 64gbps (8x 8gbps pcie3 lanes). You won't get 80 from both ports.
So I updated the firmware and I cannot get carrier light from the card. I confirmed port 1 is in ethernet mode. I am using AOI QSFP+ modules and trying to connect to my 10gb Ubiquiti switch. The manual says it should be able to run at 10gb. The OS (XCP-NG) shows the card as recognized. Could this be incompatible QSFP+ modules?
 

Rttg

Member
May 21, 2020
71
47
18
Could this be incompatible QSFP+ modules?
It’s more likely they’re stuck in VPI/IB mode (I’ve had to manually switch mine over to ethernet mode). I haven’t had any transceiver compatibility issues with the Mellanox card itself, but it’s also possible that your Ubiquiti switch is picky
 

Jamy

Member
Mar 29, 2017
44
6
8
48
It’s more likely they’re stuck in VPI/IB mode (I’ve had to manually switch mine over to ethernet mode). I haven’t had any transceiver compatibility issues with the Mellanox card itself, but it’s also possible that your Ubiquiti switch is picky
Mlxconfig shows both ports in ETH mode.


Code:
Device #1:
----------

Device type:    ConnectX3
Device:         /dev/mst/mt4099_pci_cr0

Configurations:                                      Next Boot
        SRIOV_EN                                    False(0)
        NUM_OF_VFS                                  8
        LINK_TYPE_P1                                ETH(2)
        LINK_TYPE_P2                                ETH(2)
        LOG_BAR_SIZE                                3
        BOOT_PKEY_P1                                0
        BOOT_PKEY_P2                                0
        BOOT_OPTION_ROM_EN_P1                       False(0)
        BOOT_VLAN_EN_P1                             False(0)
        BOOT_RETRY_CNT_P1                           0
        LEGACY_BOOT_PROTOCOL_P1                     None(0)
        BOOT_VLAN_P1                                1
        BOOT_OPTION_ROM_EN_P2                       False(0)
        BOOT_VLAN_EN_P2                             False(0)
        BOOT_RETRY_CNT_P2                           0
        LEGACY_BOOT_PROTOCOL_P2                     None(0)
        BOOT_VLAN_P2                                1
        IP_VER_P1                                   IPv4(0)
        IP_VER_P2                                   IPv4(0)
        CQ_TIMESTAMP                                True(1)

I've tried the transceivers and cables on another NIC and they are both good. I however cannot get any carrier light on the Mellanox card. Do I have a bad card? Anything else I should try?
 
Last edited: