ICX6650 connecting ConnectX-3 and ConnectX-3 Pro with Mellanox DAC cables only works on 1 and sometimes 2 ports on the ICX6650 40GbE rear ports.
I've read and researched for hours trying to find a solution and notice other people having similar issues with the Mellanox ConnectX-3 and ConnectX-3 Pro adapters working on all the 40GbE ports of the ICX6650, but no definitive solutions. I have both the ConnectX-3 and the Pro version adapters...they are both cross flashed HPE cards (649281-B21 and 764284-B21). I have (2) ICX6650 switches and they both act the same way. At best, only 2 of the 4 ports work at a time when the adapters are plugged in to each switch. I have tried with 2 different model Mellanox DACs (MC2207130-002 and MC2206128-005) and only ports 1/2/1 and 1/2/4 on switch 1 link and only ports 1/2/2 and 1/2/4 on switch 2 will link. That never changes no matter how many times the switches are power cycled or the servers are rebooted. Currently I am using the ConnectX-3 Pro card and have it set to Eth on both ports. If I connect DAC cables between both switches on all 4 of the 40GbE ports, they will all link and work perfectly fine which confirms the ports are all working properly on the switches.
So, as with many of the other post I have read, it is difficult to determine which component is not playing nice with each other.
In the interim, I have ordered Brocade optical transceivers and a few MPO fiber cables to see if that resolves the issue. It would still be nice to know if there are any DAC cables that would work properly. I figured by using genuine Mellanox DAC cables, that would avoid this issue.
If anyone has a working combination of these components, please provide what you are using.
I've read and researched for hours trying to find a solution and notice other people having similar issues with the Mellanox ConnectX-3 and ConnectX-3 Pro adapters working on all the 40GbE ports of the ICX6650, but no definitive solutions. I have both the ConnectX-3 and the Pro version adapters...they are both cross flashed HPE cards (649281-B21 and 764284-B21). I have (2) ICX6650 switches and they both act the same way. At best, only 2 of the 4 ports work at a time when the adapters are plugged in to each switch. I have tried with 2 different model Mellanox DACs (MC2207130-002 and MC2206128-005) and only ports 1/2/1 and 1/2/4 on switch 1 link and only ports 1/2/2 and 1/2/4 on switch 2 will link. That never changes no matter how many times the switches are power cycled or the servers are rebooted. Currently I am using the ConnectX-3 Pro card and have it set to Eth on both ports. If I connect DAC cables between both switches on all 4 of the 40GbE ports, they will all link and work perfectly fine which confirms the ports are all working properly on the switches.
So, as with many of the other post I have read, it is difficult to determine which component is not playing nice with each other.
In the interim, I have ordered Brocade optical transceivers and a few MPO fiber cables to see if that resolves the issue. It would still be nice to know if there are any DAC cables that would work properly. I figured by using genuine Mellanox DAC cables, that would avoid this issue.
If anyone has a working combination of these components, please provide what you are using.