Hey,
I have a Single Mode LC link which needs to transport 40G Infiniband. That means 40GBase-LR transceivers are my only option and I cannot use 56G MTP SR or 40G SR transceivers.
But now for the issue:
I am using ConnectX-5 cards (Dual Port) with the latest firmware, 16.32.1010 and I just can't get 40G IB to work.
When I plug in 40G LR transceivers, the card always negotiates 10G SDR
I have tried different vendors and different programmings (fs.com, Finisar) and even bought genuine Mellanox LR4 transceivers (MC2210511-LR4) out of pure despair... But no chance.
MC2210511-LR4:
10G in IB Mode, 40G in Eth mode
However, this is not a general limitation.
If I plug in a DAC
I am seeing the same limitations with some 100G transceivers
Kaiam XQX5170 (100GBase-CWDM4) -> No connection at all in IB mode, 100G in Ethernet
fs.com 100GBase-CWDM4 -> 10G in IB Mode, 100G in Ethernet
Any ideas on how to fix this?
It also bugs me that apparantly standard 100G transceivers do not work with Infiniband...
I have a Single Mode LC link which needs to transport 40G Infiniband. That means 40GBase-LR transceivers are my only option and I cannot use 56G MTP SR or 40G SR transceivers.
But now for the issue:
I am using ConnectX-5 cards (Dual Port) with the latest firmware, 16.32.1010 and I just can't get 40G IB to work.
When I plug in 40G LR transceivers, the card always negotiates 10G SDR
I have tried different vendors and different programmings (fs.com, Finisar) and even bought genuine Mellanox LR4 transceivers (MC2210511-LR4) out of pure despair... But no chance.
MC2210511-LR4:
Code:
Infiniband device 'mlx5_0' port 1 status:
default gid: fe80:0000:0000:0000:0c42:a103:006c:0170
base lid: 0xffff
sm lid: 0x0
state: 2: INIT
phys state: 5: LinkUp
rate: 10 Gb/sec (4X SDR)
link_layer: InfiniBand
Infiniband device 'mlx5_1' port 1 status:
default gid: fe80:0000:0000:0000:0c42:a103:006c:0171
base lid: 0xffff
sm lid: 0x0
state: 2: INIT
phys state: 5: LinkUp
rate: 10 Gb/sec (4X SDR)
link_layer: InfiniBand
However, this is not a general limitation.
If I plug in a DAC
Code:
Infiniband device 'mlx5_0' port 1 status:
default gid: fe80:0000:0000:0000:0c42:a103:006c:0170
base lid: 0xffff
sm lid: 0x0
state: 2: INIT
phys state: 5: LinkUp
rate: 100 Gb/sec (4X EDR)
link_layer: InfiniBand
Infiniband device 'mlx5_1' port 1 status:
default gid: fe80:0000:0000:0000:0c42:a103:006c:0171
base lid: 0xffff
sm lid: 0x0
state: 2: INIT
phys state: 5: LinkUp
rate: 100 Gb/sec (4X EDR)
link_layer: InfiniBand
Kaiam XQX5170 (100GBase-CWDM4) -> No connection at all in IB mode, 100G in Ethernet
Code:
Infiniband device 'mlx5_0' port 1 status:
default gid: fe80:0000:0000:0000:0c42:a103:006c:0170
base lid: 0xffff
sm lid: 0x0
state: 1: DOWN
phys state: 2: Polling
rate: 10 Gb/sec (4X SDR)
link_layer: InfiniBand
Infiniband device 'mlx5_1' port 1 status:
default gid: fe80:0000:0000:0000:0c42:a103:006c:0171
base lid: 0xffff
sm lid: 0x0
state: 1: DOWN
phys state: 2: Polling
rate: 10 Gb/sec (4X SDR)
link_layer: InfiniBand
Code:
Infiniband device 'mlx5_0' port 1 status:
default gid: fe80:0000:0000:0000:0c42:a103:006c:0170
base lid: 0xffff
sm lid: 0x0
state: 2: INIT
phys state: 5: LinkUp
rate: 10 Gb/sec (4X SDR)
link_layer: InfiniBand
Infiniband device 'mlx5_1' port 1 status:
default gid: fe80:0000:0000:0000:0c42:a103:006c:0171
base lid: 0xffff
sm lid: 0x0
state: 2: INIT
phys state: 5: LinkUp
rate: 10 Gb/sec (4X SDR)
link_layer: InfiniBand
It also bugs me that apparantly standard 100G transceivers do not work with Infiniband...