Direct attach with MCX354A-FCBT

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

rconan

New Member
Sep 13, 2020
3
0
1
Hi,

I have three MCX354A-FCBT adapters I wanted to use for "full mesh" ethernet connectivity between three servers but I don't seem to be getting links.
I have the cards configured in ethernet mode:
Code:
# mlxconfig -d /dev/mst/mt4099_pciconf0 q

Device #1:
----------

Device type:    ConnectX3
Device:         /dev/mst/mt4099_pciconf0

Configurations:                              Next Boot
         SRIOV_EN                            False(0)
         NUM_OF_VFS                          8
         LINK_TYPE_P1                        ETH(2)
         LINK_TYPE_P2                        ETH(2)
         LOG_BAR_SIZE                        3
         BOOT_PKEY_P1                        0
         BOOT_PKEY_P2                        0
         BOOT_OPTION_ROM_EN_P1               True(1)
         BOOT_VLAN_EN_P1                     False(0)
         BOOT_RETRY_CNT_P1                   0
         LEGACY_BOOT_PROTOCOL_P1             PXE(1)
         BOOT_VLAN_P1                        1
         BOOT_OPTION_ROM_EN_P2               True(1)
         BOOT_VLAN_EN_P2                     False(0)
         BOOT_RETRY_CNT_P2                   0
         LEGACY_BOOT_PROTOCOL_P2             PXE(1)
         BOOT_VLAN_P2                        1
         IP_VER_P1                           IPv4(0)
         IP_VER_P2                           IPv4(0)
         CQ_TIMESTAMP                        True(1)
I am able to see the interfaces and supported link types
Code:
# ethtool enp3s0d1
Settings for enp3s0d1:
        Supported ports: [ FIBRE ]
        Supported link modes:   1000baseKX/Full
                                10000baseKX4/Full
                                10000baseKR/Full
                                40000baseCR4/Full
                                40000baseSR4/Full
                                56000baseCR4/Full
                                56000baseSR4/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  1000baseKX/Full
                                10000baseKX4/Full
                                10000baseKR/Full
                                40000baseCR4/Full
                                40000baseSR4/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: Unknown!
        Duplex: Unknown! (255)
        Port: FIBRE
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: off
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000014 (20)
                               link ifdown
        Link detected: no
They are connected using passive DACs, specifically Mellanox MC2206130-00A 40G QSFP+ DAC Cable

But the link is not up
Code:
# ip l l enp3s0d1
7: enp3s0d1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether e4:1d:2d:1c:38:22 brd ff:ff:ff:ff:ff:ff
My best guess is that these cannot be linked like this using passive DACs?
 

rconan

New Member
Sep 13, 2020
3
0
1
Having done a little research it seems likely that these adapters cannot be linked with passive cables.
Does anyone know if they have requirements on the coding of the QSFPs?
 

Netwerkz101

Active Member
Dec 27, 2015
308
90
28
Having done a little research it seems likely that these adapters cannot be linked with passive cables.
Does anyone know if they have requirements on the coding of the QSFPs?
I use the exact same cable from FS.com.
They seem to work fine for me.

Have you flashed your cards at all?
Are your cards plugged directly into a x8 PCIe slot?
Have you tried just connecting two ports on the same card together with the passive DAC cable?
 

rconan

New Member
Sep 13, 2020
3
0
1
They are cross-flashed from MCX354A-QCBT
they're plugged into one of the PCIe slots in a Dell R720
I will give the two ports on the same card a go
 

necr

Active Member
Dec 27, 2017
151
47
28
124
Hmm, normally DAC is recognised as Direct Attach Copper and Auto-Negotiation on is set. I would also try IB and a different firmware.
 

i386

Well-Known Member
Mar 18, 2016
4,220
1,540
113
34
Germany
My best guess is that these cannot be linked like this using passive DACs?
I'm using different broadcom qsfp+ passive dac cables here without problems.


Edit: I just saw that you linked to fs.com. In the last two months there werde different posts that the fs.com cables did not work (lack of quality?).
Try some cables from different brands (broadcom, ibm etc.)
 

Roelf Zomerman

Active Member
Jan 10, 2019
147
27
28
blog.azureinfra.com
I have fs.com cables and got direct attached working.. now the thing is that when my NIC's where in AUTODETECT mode, the passive cables did not do anything.. but when I set the hardware NIC port to ETH (mode 2) they worked flawlessly..

I got the Customized 40GQSFP+ Passive Direct Attach Copper Cable #69907 with Cisco coding on one end, Broccade on the other end (doesnt matter as long as it isnt Mellanox coding [I only got 10G on those] )

And you can set the NIC to ETH using the following post (linux): https://forums.servethehome.com/ind...rnet-dual-port-qsfp-adapter.20525/post-198015 or Windows (https://forums.servethehome.com/ind...rnet-dual-port-qsfp-adapter.20525/post-279534)

And I have them in 2x Dell R620's