ICX6610 40gbe breakout port to 40gbe NIC

lizardking009

Member
Oct 16, 2013
38
10
8
A few weeks ago, I attempted to connect a 40gbe ConnectX-3 Pro NIC to an ICX6610 40gbe breakout port with a QSFP DAC. The port connected but only one link at 10gb. Moving to one of the 40gbe nonbreakout port linked at 40gb.

Is there a QSFP NIC that will see all 4x10gb ports and allow me to bond them within the OS?

Or did I screw up on configuring the breakout port?
 

kapone

Well-Known Member
May 23, 2015
1,001
567
113
I may be going blind...where does it say it supports breakout cables?? And that it will "bond" them into a single connection?
 

lizardking009

Member
Oct 16, 2013
38
10
8
First page, end of the first big paragraph.

"The 40GbE QSFP+ adapter supports an optical 40GBASE-SR4 connection or copper 40GBASE-CR4 connection which can be re-configured to support 4-ports of 10GBASE-CR for use with a QSFP+ to quad breakout SFP+ direct attach cable (DAC) or Active Optical Cable (AOC)"

It won't bond them but I can do that on the OS. I ordered one - we'll see if I can do this with a QSFP DAC.
 
  • Like
Reactions: kapone

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,177
1,655
113
CA
First page, end of the first big paragraph.

"The 40GbE QSFP+ adapter supports an optical 40GBASE-SR4 connection or copper 40GBASE-CR4 connection which can be re-configured to support 4-ports of 10GBASE-CR for use with a QSFP+ to quad breakout SFP+ direct attach cable (DAC) or Active Optical Cable (AOC)"

It won't bond them but I can do that on the OS. I ordered one - we'll see if I can do this with a QSFP DAC.
That supports the same function as the 6610 40gig
right? Not the other way?
 

lizardking009

Member
Oct 16, 2013
38
10
8
That supports the same function as the 6610 40gig
right? Not the other way?
It looks like it supports breakout on the NIC side. My hope is that if I connect a QSFP DAC with it and the ICX6610 breakout, I'll get 4 lanes of 10gb that I can then bond together in the OS.
 

lizardking009

Member
Oct 16, 2013
38
10
8
I got my 40gb DAC (10gtek) and the Emulex card. I updated firmware to latest. The ICX saw 4 10gb ports up, but the Emulex reports only a 40gb port. I didn't find any knobs or documentation to tell it to use breakout mode. There is multichannel mode but that isn't the same thing.

I do have a 40gb MTP MM SFP and a fiber breakout cable. I'll see if that's any different.

What I really want is a switch with 4x40gb ports and 16x10gb ports at a Mikrotik price point. I can dream.
 

ServerUser

New Member
Aug 29, 2020
5
2
3
Is there a QSFP NIC that will see all 4x10gb ports and allow me to bond them within the OS?
Yes, in a fashion, the Intel XL710 will allow you to use a breakout cable to get 4 x 10GE ports and then bond in the OS - but that gives you 4 x SFP+ ports at one end. I haven't tried it with QSFP at both ends to see if you can get 4x10GE over a QSFP. Unfortunately I don't have one to hand at the moment to test, I have a feeling it may work though.

You need to use the Intel QSFP Configuration Tool through to tell the NIC to use 4x10 configuration. If you've got a Dual Port card you can do 2x2x10 and get 2x10GE on both ports. As these cards are PCI-e x8, Intel limits them to 40GE total throughput so you can't do 2x4x10.
 

klui

Active Member
Feb 3, 2019
291
123
43
I got my 40gb DAC (10gtek) and the Emulex card. I updated firmware to latest. The ICX saw 4 10gb ports up, but the Emulex reports only a 40gb port. I didn't find any knobs or documentation to tell it to use breakout mode. There is multichannel mode but that isn't the same thing.

I do have a 40gb MTP MM SFP and a fiber breakout cable. I'll see if that's any different.

What I really want is a switch with 4x40gb ports and 16x10gb ports at a Mikrotik price point. I can dream.
I think there is a misunderstanding between the datasheet and the eBay listing. The datasheet talks about the OCe14000B cards and the eBay listing is for non-B card. The 2015 manual for OCe14000 series do not mention breaking out 4 SFP+ ports from a QSFP+ breakout originating from the card.

The installation manual https://www.mouser.com/datasheet/2/678/ce14000_manual_elx-1112297.pdf for the 14000B also does not mention breakouts either. The datasheet was probably referring to the 4-port SFP+ variant. Typically if a card supports splitting QSFP ports, there should be at least 4 LEDs and these only have 2.