ICX6610 40gbe breakout port to 40gbe NIC

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

lizardking009

Member
Oct 16, 2013
38
10
8
A few weeks ago, I attempted to connect a 40gbe ConnectX-3 Pro NIC to an ICX6610 40gbe breakout port with a QSFP DAC. The port connected but only one link at 10gb. Moving to one of the 40gbe nonbreakout port linked at 40gb.

Is there a QSFP NIC that will see all 4x10gb ports and allow me to bond them within the OS?

Or did I screw up on configuring the breakout port?
 

lizardking009

Member
Oct 16, 2013
38
10
8
  • Like
Reactions: klui

kapone

Well-Known Member
May 23, 2015
1,095
642
113
I may be going blind...where does it say it supports breakout cables?? And that it will "bond" them into a single connection?
 

lizardking009

Member
Oct 16, 2013
38
10
8
First page, end of the first big paragraph.

"The 40GbE QSFP+ adapter supports an optical 40GBASE-SR4 connection or copper 40GBASE-CR4 connection which can be re-configured to support 4-ports of 10GBASE-CR for use with a QSFP+ to quad breakout SFP+ direct attach cable (DAC) or Active Optical Cable (AOC)"

It won't bond them but I can do that on the OS. I ordered one - we'll see if I can do this with a QSFP DAC.
 
  • Like
Reactions: kapone

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,625
2,043
113
First page, end of the first big paragraph.

"The 40GbE QSFP+ adapter supports an optical 40GBASE-SR4 connection or copper 40GBASE-CR4 connection which can be re-configured to support 4-ports of 10GBASE-CR for use with a QSFP+ to quad breakout SFP+ direct attach cable (DAC) or Active Optical Cable (AOC)"

It won't bond them but I can do that on the OS. I ordered one - we'll see if I can do this with a QSFP DAC.
That supports the same function as the 6610 40gig
right? Not the other way?
 

lizardking009

Member
Oct 16, 2013
38
10
8
That supports the same function as the 6610 40gig
right? Not the other way?
It looks like it supports breakout on the NIC side. My hope is that if I connect a QSFP DAC with it and the ICX6610 breakout, I'll get 4 lanes of 10gb that I can then bond together in the OS.
 

lizardking009

Member
Oct 16, 2013
38
10
8
I got my 40gb DAC (10gtek) and the Emulex card. I updated firmware to latest. The ICX saw 4 10gb ports up, but the Emulex reports only a 40gb port. I didn't find any knobs or documentation to tell it to use breakout mode. There is multichannel mode but that isn't the same thing.

I do have a 40gb MTP MM SFP and a fiber breakout cable. I'll see if that's any different.

What I really want is a switch with 4x40gb ports and 16x10gb ports at a Mikrotik price point. I can dream.
 

ServerUser

New Member
Aug 29, 2020
5
2
3
Is there a QSFP NIC that will see all 4x10gb ports and allow me to bond them within the OS?
Yes, in a fashion, the Intel XL710 will allow you to use a breakout cable to get 4 x 10GE ports and then bond in the OS - but that gives you 4 x SFP+ ports at one end. I haven't tried it with QSFP at both ends to see if you can get 4x10GE over a QSFP. Unfortunately I don't have one to hand at the moment to test, I have a feeling it may work though.

You need to use the Intel QSFP Configuration Tool through to tell the NIC to use 4x10 configuration. If you've got a Dual Port card you can do 2x2x10 and get 2x10GE on both ports. As these cards are PCI-e x8, Intel limits them to 40GE total throughput so you can't do 2x4x10.
 

klui

Well-Known Member
Feb 3, 2019
824
453
63
I got my 40gb DAC (10gtek) and the Emulex card. I updated firmware to latest. The ICX saw 4 10gb ports up, but the Emulex reports only a 40gb port. I didn't find any knobs or documentation to tell it to use breakout mode. There is multichannel mode but that isn't the same thing.

I do have a 40gb MTP MM SFP and a fiber breakout cable. I'll see if that's any different.

What I really want is a switch with 4x40gb ports and 16x10gb ports at a Mikrotik price point. I can dream.
I think there is a misunderstanding between the datasheet and the eBay listing. The datasheet talks about the OCe14000B cards and the eBay listing is for non-B card. The 2015 manual for OCe14000 series do not mention breaking out 4 SFP+ ports from a QSFP+ breakout originating from the card.

The installation manual https://www.mouser.com/datasheet/2/678/ce14000_manual_elx-1112297.pdf for the 14000B also does not mention breakouts either. The datasheet was probably referring to the 4-port SFP+ variant. Typically if a card supports splitting QSFP ports, there should be at least 4 LEDs and these only have 2.
 

rootpeer

Member
Oct 19, 2019
73
13
8
Maybe try with a QSFP breakout cable and two dual port SFP+ cards on the client machine first?
 

dodgy route

Member
Aug 12, 2020
48
64
18
Australia
I got my 40gb DAC (10gtek) and the Emulex card. I updated firmware to latest. The ICX saw 4 10gb ports up, but the Emulex reports only a 40gb port. I didn't find any knobs or documentation to tell it to use breakout mode. There is multichannel mode but that isn't the same thing.

I do have a 40gb MTP MM SFP and a fiber breakout cable. I'll see if that's any different.

What I really want is a switch with 4x40gb ports and 16x10gb ports at a Mikrotik price point. I can dream.
Do you still have this card and the 6610?

Are you able to update the firmware to latest on the card and then also flash it to change mode to 4x10 instead of 1x40?
I suspect it was an added feature, it was definitely added in 2016 firmware to OCe14401B (reference: https://docs.broadcom.com/doc/12356067), cant find the OCe14401 reference that it was added but this document clearly shows its possible on the OCe14401-UX, just searching Broadcoms site makes me want to vomit excessively.

Page 392, section 5.16.2 specifically lists this card if it is a OCe14401-UX indeed

HTML:
GetLinkConfig
This command displays the active and configured link configuration IDs and lists the available link configurations (descriptions).
NOTE The link reconfiguration feature is available on only some OCe14000-series adapters.

Supported By
Linux, Solaris, Windows, and Windows + CIM Provider on a VMware host

Syntax
GetLinkConfig <MAC|WWPN>

Parameters
WWPN The WWPN of any FCoE function on the adapter.
MACThe MAC address of any NIC+RoCE or iSCSI function on the adapter.

Example
OCM: brcmhbacmd GetLinkConfig 00-90-fa-41-28-f0

Active Link ID: 19
Configured Link ID: 19

Available Link Configurations

ID Description
-- ---------------------------------
19 1x40Gbps QSFP+ OCe14401-UX CNA
20 4x10Gbps SFP+ OCe14401-UX CNA
Also to note about cables, again in relation to OCe14401B (reference: https://docs.broadcom.com/doc/12378898)
Not sure what these are, but suspect not a problem with a DAC
Code:
Other Known Issues.
Certain Cisco breakout cables are not properly detected by the OCe14401B adapter.
The following Cisco breakout cables do not provide the correct codes to the OCe14401B adapter to support the SFF-8436 standard:
QSFP-4X10G-AOC10M
QSFP-4X10G-AOC7M
QSFP-4X10G-AOC5M
QSFP-4X10G-AOC3M
QSFP-4X10G-AOC2M
QSFP-4X10G-AOC1M

If you are using one of these cables to connect an OCe14401B adapter to a Cisco Nexus-series switch, the following error is reported:
Optics faulted/incorrectly installed/not installed -Reseat optics, if issue not resolved, replace. Link is Non Operational

Workaround
Use Arista, Brocade, or International Business Machines (IBM) breakout cables with OCe14401B adapters.
 
Last edited:

clcorbin

Member
Feb 15, 2014
38
7
8
Do we have a final ruling on this? I was hoping to come up with configuration that would allow me to use the breakout port on my ICX6610 to connect to a QSFP connector in "some" NIC and show up as 4 x 10Gb ports on the server. Preferably with a single MPO to MPO cable linking things together.
 
  • Like
Reactions: jasonsansone

clcorbin

Member
Feb 15, 2014
38
7
8
I was digging into the Intel XL710-QDA1(2) and it lists 4x10Gb breakout as one of it's operating modes. It is a bit expensive, so I'm not in TOO big a hurry to buy one until I know it CAN link up with the breakout port on the switch and bring in 4 10Gb links to my server.
 

LemonheadST

New Member
Aug 25, 2022
5
4
3
I was digging into the Intel XL710-QDA1(2) and it lists 4x10Gb breakout as one of it's operating modes. It is a bit expensive, so I'm not in TOO big a hurry to buy one until I know it CAN link up with the breakout port on the switch and bring in 4 10Gb links to my server.
I know this thread is kind of old...however I have a XL710-QDA1 NIC and can confirm that it will link up fine to the 4x10 QSFP+ ports on the ICX6610 fine. You need to use the Intel QSFP+ utility to set the NIC to 4x10 mode (the default is 1x40).
On the host side it presents 4 usable ethernet interfaces at 10gb each, which can be combined into a 802.3ad bond for a 40gb link.
I'm using MPO fiber QSFP+ transievers on both ends.
Hope this helps!
 

Rootless

Thank you
Aug 9, 2021
17
7
3
I know this thread is kind of old...however I have a XL710-QDA1 NIC and can confirm that it will link up fine to the 4x10 QSFP+ ports on the ICX6610 fine. You need to use the Intel QSFP+ utility to set the NIC to 4x10 mode (the default is 1x40).
On the host side it presents 4 usable ethernet interfaces at 10gb each, which can be combined into a 802.3ad bond for a 40gb link.
I'm using MPO fiber QSFP+ transievers on both ends.
Hope this helps!
I'm trying to do this same thing.

What QSFP+ transceivers are you using?

Is there an LC - LC version? From what I read about MPO, it seems a little overkill for this and much pricier.

The XL710-QDA1 is a little pricey. Is that my only option for 40Gb using the bonded channels needed by the ICX6610?

Thank you for responding to the thread in September.
 

LodeRunner

Active Member
Apr 27, 2019
540
227
43
Is there an LC - LC version? From what I read about MPO, it seems a little overkill for this and much pricier.
40Gb over LC requires a BiDi transceiver. The standard is MPO/MTP because 40Gb is really 4x 10Gb channels spread over 4 pairs. The BiDi smashes this into a single pair by doing DWDM (or is it CWDM?) to do multiple light frequencies over a single fiber.
 
  • Like
Reactions: mach3.2