MellanoxX-3 + Cisco 3064 - 40G

[Roelf Zomerman]

HI everyone,

I'm trying to connect my servers to my new cisco switch.. the switch itself is set to 40G, but so far.. no link.. no link .. no link..

I'm using direct DAC cables from FS QSFP-PC01 40G 1meter

Now the weird thing is that if I make a loop (ConnectX-3 Port1->Port2) with the same cable it works and I have a link but only at 10G.. If I make a loop on my switch with that cable it works (40G)..


How can I set my Mellanox to 40G ? It's a CX354A-FCBT Rev A4

The switch reports:
Ethernet1/49 is down (Link not connected)
admin state is up, Dedicated Interface
Hardware: 40000 Ethernet, address: 7c0e.cefc.e038 (bia 7c0e.cefc.e038)
MTU 1500 bytes, BW 40000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is access
auto-duplex, auto-speed, media type is 40G


My Mellanox is in Ethernet mode and I have updated it with the latest flash guide (https://forums.servethehome.com/ind...net-dual-port-qsfp-adapter.20525/#post-198015)

mlxfwmanager --query
Querying Mellanox devices firmware ...

Device #1:
----------

Device Type: ConnectX3
Part Number: MCX354A-FCB_A2-A5
Description: ConnectX-3 VPI adapter card; dual-port QSFP; FDR IB (56Gb/s) and 40GigE; PCIe3.0 x8 8GT/s; RoHS R6
PSID: MT_1090120019
PCI Device Name: mt4099_pci_cr0
Port1 MAC: f452147c1f71
Port2 MAC: f452147c1f72
Versions: Current Available
FW 2.42.5000 2.42.5000
PXE 3.4.0752 3.4.0752

Status: Up to date



:mlxconfig -d /dev/mst/mt4099_pci_cr0 query
Device #1:
----------

Device type: ConnectX3
Device: /dev/mst/mt4099_pci_cr0

Configurations: Next Boot
SRIOV_EN False(0)
NUM_OF_VFS 8
LINK_TYPE_P1 ETH(2)
LINK_TYPE_P2 ETH(2)
LOG_BAR_SIZE 3
BOOT_PKEY_P1 0
BOOT_PKEY_P2 0
BOOT_OPTION_ROM_EN_P1 False(0)
BOOT_VLAN_EN_P1 False(0)
BOOT_RETRY_CNT_P1 0
LEGACY_BOOT_PROTOCOL_P1 None(0)
BOOT_VLAN_P1 1
BOOT_OPTION_ROM_EN_P2 False(0)
BOOT_VLAN_EN_P2 False(0)
BOOT_RETRY_CNT_P2 0
LEGACY_BOOT_PROTOCOL_P2 None(0)
BOOT_VLAN_P2 1
IP_VER_P1 IPv4(0)
IP_VER_P2 IPv4(0)
CQ_TIMESTAMP True(1)


Any ideas?

 

[klui]

Manually set your CX3 to 40G. If you still don't get a link, set your Cisco's port to 40G and if it has FEC, enable it.

 

[Roelf Zomerman]

Manually set your CX3 to 40G. If you still don't get a link, set your Cisco's port to 40G and if it has FEC, enable it.

But how? I can see through PowerShell that the link state is set to not-auto .. but I have no way to change it..

PS C:\Users\Administrator> Get-MlnxPCIDeviceCapabilities


Caption : MLNX_PCIDeviceCapabilities 'Mellanox ConnectX-3 VPI (MT04099) Network Adapter'
Description : Mellanox ConnectX-3 VPI (MT04099) Network Adapter
ElementName : HCA 0
InstanceID : PCI\VEN_15B3&DEV_1003&SUBSYS_005015B3&REV_00\F4521403007C1F7000
DualPort : True
Name : HCA 0
PortOneAutoSense : False
PortOneDefault : True
PortOneDoSenseAllowed : True
PortOneEth : True
PortOneIb : False
PortTwoAutoSenseCap : False
PortTwoDefault : True
PortTwoDoSenseAllowed : True
PortTwoEth : True
PortTwoIb : False
SystemName : WIN-HOLRCO5EE37
PSComputerName :

 

[i386]

I'm using direct DAC cables from FS QSFP-PC01 40G 1meter

I think it's the fs.com cable...
What happens if you use an actual cisco (or any other known brand) dac?

 

[klui]

But how?

Device manager

 

[Roelf Zomerman]

Device manager

There are a lot of options in there.. but Link Speed, Speed, or anything like it is not there..


Flow Control
Interrupt Moderation
IPV4 Checksum Offload
Jumbo Packet
Large Send Offload V2 (IPv4)
Large Send Offload V2 (IPv6)
Maximum number of RSS Processors
NetworkDirect Functionality
Preferred NUMA node
Maximum Number of RSS Queues
PacketDirect Functionality
Priority & Vlan Tag
Quality Of Service
Receive Buffers
Recv Segment Coalescing (IPv4)
Recv Segment Coalescing (IPv6)
Receive Side Scaling
RSS Base Processor Number
Virtual Switch RSS
RSS load balancing Profile
SR-IOV
TCP/UDP Checksum Offload (IPv4)
TCP/UDP Checksum Offload (IPv6)
Send Buffers
Virtual Machine Queues
VMQ VLAN Filtering
Ignore FCS errors
Locally Administered Address
Transmit Control Blocks
Receive Completion Method
R/RoCE Max Frame Size
Rx Buffer Alignment
Rx Interrupt Moderation Type
Rx Interrupt Moderation Profile
Number of Polls on Receive
Tx Throughput Port Arbiter
Tx Interrupt Moderation Profile
VLAN ID

 

[klui]

You're right. I saw the setting when using an older version of the Windows OFED driver. The latest version of WinMFT mlxlink says the device is no longer supported. You may want to double check using Linux and use ethtool or an older version of the Windows driver.

@i386 might be on to something, too. I used a Mellanox AOC/DAC in loopback mode and the ports link up in 40G. These DACs also work in 100G. Try setting your Cisco to 10G and see if it links up using your FS cable.

EDIT: Try as I might under Windows, I couldn't manually set speed. Under Win10 2004 the older driver won't even install. It's possible that version (....52000) was the same base version in 2004. I had to update to a more recent version to get VLAN functionality. ethtool under Linux works fine and I can manually set speed and my AOC will link at 10G for example. If I don't it links at 56G.

 

[Roelf Zomerman]

tested with 2 other cables and got the 40G speeds on the local card...

Brocade 58-000033-01

And

Brocade 58-000041-01 (active QSFP)