ICX6610 40gbe breakout port to 40gbe NIC

[LemonheadST]

I'm trying to do this same thing.

What QSFP+ transceivers are you using?

Is there an LC - LC version? From what I read about MPO, it seems a little overkill for this and much pricier.

The XL710-QDA1 is a little pricey. Is that my only option for 40Gb using the bonded channels needed by the ICX6610?

Thank you for responding to the thread in September.

I'm just using 10GTek QSFP+ MPO transceivers from Amazon
Besides MPO, if you can just do a short run (5m or less) a DAC cable like this one should be cheaper.
I don't know of any other cards that support 4x10 mode besides the XL710 - in fact I bought one after seeing this thread.

 

[Freebsd1976]

buy this card today, good news and bad news.
good news is it has lot feature (roce2 sr-iov iscsi foce), bad news is it don't support port split .
repeat!!! the OCe14401-UX does not support split to 4x10G, tested today, flash 11.x from boardcom and 12.x from lenovo ,both only one port 40G( and it also works on 10G when used a qsfp to sfp and cisco 3m DAC ), if you need port split to 4x10G ,don't buy it.

# hbacmd GetLinkConfig 00-90-fa-xx-xx-xx

ERROR: <251>: Hardware or firmware does not support command.

Do you still have this card and the 6610?

Are you able to update the firmware to latest on the card and then also flash it to change mode to 4x10 instead of 1x40?
I suspect it was an added feature, it was definitely added in 2016 firmware to OCe14401B (reference: https://docs.broadcom.com/doc/12356067), cant find the OCe14401 reference that it was added but this document clearly shows its possible on the OCe14401-UX, just searching Broadcoms site makes me want to vomit excessively.

https://docs.broadcom.com/doc/12357302

Page 392, section 5.16.2 specifically lists this card if it is a OCe14401-UX indeed

GetLinkConfig
This command displays the active and configured link configuration IDs and lists the available link configurations (descriptions).
NOTE The link reconfiguration feature is available on only some OCe14000-series adapters.

Supported By
Linux, Solaris, Windows, and Windows + CIM Provider on a VMware host

Syntax
GetLinkConfig <MAC|WWPN>

Parameters
WWPN The WWPN of any FCoE function on the adapter.
MACThe MAC address of any NIC+RoCE or iSCSI function on the adapter.

Example
OCM: brcmhbacmd GetLinkConfig 00-90-fa-41-28-f0

Active Link ID: 19
Configured Link ID: 19

Available Link Configurations

ID Description
-- ---------------------------------
19 1x40Gbps QSFP+ OCe14401-UX CNA
20 4x10Gbps SFP+ OCe14401-UX CNA

Also to note about cables, again in relation to OCe14401B (reference: https://docs.broadcom.com/doc/12378898)
Not sure what these are, but suspect not a problem with a DAC

Other Known Issues.
Certain Cisco breakout cables are not properly detected by the OCe14401B adapter.
The following Cisco breakout cables do not provide the correct codes to the OCe14401B adapter to support the SFF-8436 standard:
QSFP-4X10G-AOC10M
QSFP-4X10G-AOC7M
QSFP-4X10G-AOC5M
QSFP-4X10G-AOC3M
QSFP-4X10G-AOC2M
QSFP-4X10G-AOC1M

If you are using one of these cables to connect an OCe14401B adapter to a Cisco Nexus-series switch, the following error is reported:
Optics faulted/incorrectly installed/not installed -Reseat optics, if issue not resolved, replace. Link is Non Operational

Workaround
Use Arista, Brocade, or International Business Machines (IBM) breakout cables with OCe14401B adapters.

 

[ramicio]

I know this thread is kind of old...however I have a XL710-QDA1 NIC and can confirm that it will link up fine to the 4x10 QSFP+ ports on the ICX6610 fine. You need to use the Intel QSFP+ utility to set the NIC to 4x10 mode (the default is 1x40).
On the host side it presents 4 usable ethernet interfaces at 10gb each, which can be combined into a 802.3ad bond for a 40gb link.
I'm using MPO fiber QSFP+ transievers on both ends.
Hope this helps!

Hello. Can the XL710 link up to the 1/2/1 or 1/2/6 ports at 1x40g? I'm having major issues with trying this. I can connect fine to the breakout ports (switch side) with a QSFP+ DAC. But I want a 40g link, not four 10g links LAGed and capped to 10g.

 

[colinb]

Hello. Can the XL710 link up to the 1/2/1 or 1/2/6 ports at 1x40g? I'm having major issues with trying this. I can connect fine to the breakout ports (switch side) with a QSFP+ DAC. But I want a 40g link, not four 10g links LAGed and capped to 10g.

Did you get an answer on this? I'm in the same place, having one or both of the 6610's QSFP breakout ports function as "pure" 40G single link would make things much easier!

 

[NablaSquaredG]

Did you get an answer on this? I'm in the same place, having one or both of the 6610's QSFP breakout ports function as "pure" 40G single link would make things much easier!

No, you can't. 2 of the 40G ports are "true" 40G ports, whereas the other 2 40G ports are always in breakout mode. You can't change that.

LAG and a 4x10G capable NIC will be your only option

 

[colinb]

No, you can't. 2 of the 40G ports are "true" 40G ports, whereas the other 2 40G ports are always in breakout mode. You can't change that.

LAG and a 4x10G capable NIC will be your only option

Cheers. Will plan accordingly!

 

[jasonsansone]

FYI I just did this based on this thread. Thanks to everyone for the tips. I am using cheap CX3 NICs on the 2x 40Gbe ports and I installed a xl710-DA2 on the third server attached to the 40Gbe breakout port. All using same Mellanox DAC’s. Updated the Intel NVM and set to 4x10. Worked like a charm.

 

[ramicio]

FYI I just did this based on this thread. Thanks to everyone for the tips. I am using cheap CX3 NICs on the 2x 40Gbe ports and I installed a xl710-DA2 on the third server attached to the 40Gbe breakout port. All using same Mellanox DAC’s. Updated the Intel NVM and set to 4x10. Worked like a charm.

My question was about the two 40-gig non-breakout ports. I couldn't get the XL710 to work with them (1x40). I could only get it to work on the breakout ports at 4x10.

 

[jasonsansone]

My question was about the two 40-gig non-breakout ports. I couldn't get the XL710 to work with them (1x40). I could only get it to work on the breakout ports at 4x10.

did you follow the steps to make sure those ports are not still setup as stacking ports? I didn’t have to do any further config. My 2x 40gbe ports connected at 40gbe links without issues.

 

[ramicio]

did you follow the steps to make sure those ports are not still setup as stacking ports? I didn’t have to do any further config. My 2x 40gbe ports connected at 40gbe links without issues.

Yes. I had help from others who looked at my config and had me do all kinds of tests. Linux wouldn't grab an IP. Had to make it static. Windows would grab an IP. In either situation the connection was unusable. Linux specifically showed they were ethernet frame errors. Tried 2 different cards, and both ports acted the same. Cards connected fine together (server to server). Bought two CX3 cards and it all works. But now I'm stuck with over $500 of cards I can't use.

 

[jasonsansone]

Yes. I had help from others who looked at my config and had me do all kinds of tests. Linux wouldn't grab an IP. Had to make it static. Windows would grab an IP. In either situation the connection was unusable. Linux specifically showed they were ethernet frame errors. Tried 2 different cards, and both ports acted the same. Cards connected fine together (server to server). Bought two CX3 cards and it all works. But now I'm stuck with over $500 of cards I can't use.

Updated to newest NVM? Don't have many other ideas.

 

[ramicio]

Updated to newest NVM? Don't have many other ideas.

Yes I was on 27.7. Tried multiple DAC cables. Something doesn't like each other. Tried changing a crapload of hardware settings under ethtool too.