ICX6650 connecting ConnectX-3 and ConnectX-3 Pro with Mellanox DAC cables not working properly

[datanet]

ICX6650 connecting ConnectX-3 and ConnectX-3 Pro with Mellanox DAC cables only works on 1 and sometimes 2 ports on the ICX6650 40GbE rear ports.

I've read and researched for hours trying to find a solution and notice other people having similar issues with the Mellanox ConnectX-3 and ConnectX-3 Pro adapters working on all the 40GbE ports of the ICX6650, but no definitive solutions. I have both the ConnectX-3 and the Pro version adapters...they are both cross flashed HPE cards (649281-B21 and 764284-B21). I have (2) ICX6650 switches and they both act the same way. At best, only 2 of the 4 ports work at a time when the adapters are plugged in to each switch. I have tried with 2 different model Mellanox DACs (MC2207130-002 and MC2206128-005) and only ports 1/2/1 and 1/2/4 on switch 1 link and only ports 1/2/2 and 1/2/4 on switch 2 will link. That never changes no matter how many times the switches are power cycled or the servers are rebooted. Currently I am using the ConnectX-3 Pro card and have it set to Eth on both ports. If I connect DAC cables between both switches on all 4 of the 40GbE ports, they will all link and work perfectly fine which confirms the ports are all working properly on the switches.

So, as with many of the other post I have read, it is difficult to determine which component is not playing nice with each other.

In the interim, I have ordered Brocade optical transceivers and a few MPO fiber cables to see if that resolves the issue. It would still be nice to know if there are any DAC cables that would work properly. I figured by using genuine Mellanox DAC cables, that would avoid this issue.

If anyone has a working combination of these components, please provide what you are using.

 

[datanet]

Update:

I received the Brocade optical transceivers and the MPO fiber patch cables and I can confirm it works on all of the 40GbE ports without any issues.

The Brocade (57-1000267-01) optical transceivers work perfectly fine in the ICX6650 and in the Mellanox ConnectX-3 adapters despite the specs showing them as FiberChannel transceivers. Everything links up on any of the 4 rear ports of the ICX6650 and all links show as 40GbE connection.

For easy reference to anyone having this issue in the future, just search for "Brocade 57-1000267-01 XBR-000232" on eBay. I paid a little under $10 for each of these transceivers.

The MPO fiber cable I used was this one from Amazon
https://www.amazon.com/gp/product/B07T1PZ6YP/ref=ppx_yo_dt_b_asin_title_o00_s00?ie=UTF8&th=1

The important thing to search for with the fiber patch cable is MPO to MPO female connector, type B polarity. You can use 8, 12, or 24 (whichever you can find cheapest for the length you need) since the transceiver only uses 8 fiber strands (4 send/4 receive - 4X10GbE).

All in, you're probably right around $50 per connection...but it works without any issues.

 

[klui]

I've never worked with MPO/MTP transceivers. Do these have pins? The cable you linked has no pins.

 

[datanet]

The transceiver has 2 alignment pins (male connector) and the MPO cable has a female connector with 2 holes for the alignment pins.

 

[xtantaudio]

I received the Brocade optical transceivers and the MPO fiber patch cables and I can confirm it works on all of the 40GbE ports without any issues

Did you ever find a DAC Cable that worked ? I am experiencing this same issue with my 6650 and would like to use DAC, but MPO fiber may just have to do as I really want 40Gb connectivity.