MTO Questions

Rdl

New Member
Jan 11, 2021
15
1
3
Just got a hold of a couple of mellanox connectx-3 network cards and have some easy questions.

1. In a nic to nic connection which type would I use, Type A, Type B, or Type C. also 8/12/24 fiber
2. In a nic to switch connection which type would be used.
3. What is the third type used for.

Thanks
 

MoMeanMugs

Member
Apr 16, 2018
32
3
8
71
I used this when I needed to understand:


I have some ConnectX-3's as well. Whether you connect NIC to NIC or NIC to switch, you want a B cable that reverses the fibers. You should only need 8 fibers (4 send at 10Gb per fiber and 4 receive at 10Gb per fiber), but I bought 12 fiber cables off eBay since the ones I found were cheap (personal seller trying to offload extras). The 12 fiber cables work fine. I bought some Brocade 40Gb QSFP's (search for Brocade 57-1000267-01 XBR-000232) and use female B to female B. Hope this helps.
 

Rdl

New Member
Jan 11, 2021
15
1
3
Thank you for your reply. I have read that a couple of times. I havn't worked with any fiber since the 1 g days so the 10 and 40 protocols are new to me. I ordered a type B cable and thought it should be the right one as per the old standards but the cards are showing they are up but I have no cable connection which had me wondering. I have also noticed that on my adapter properties all the threads I have read have show a port tab to select ether or ini mine is not showing that option, would you know it the Winof-5_50_5400_all_win2019_x64 versions have dropped that? I am still looking for the reason I'm showing a no cable connection between the two nics.

Thanks
 

MoMeanMugs

Member
Apr 16, 2018
32
3
8
71
I haven't used them under Windows, so I won't be of much help there. I use mine under FreeBSD (TrueNAS). There are Mellanox commandline tools (I forget the exact name) that you can set the ports to ETH (and save it to the EEPROM so you never have to worry about it again). I thought I had read the tool was available for Windows, but it for sure exists for Linux and FreeBSD. Maybe you can boot off a live Linux DVD and put the tool on a USB drive to check the parameters. Will look for the name of the tool, but you can find reference to it on the forums here.
 

Rdl

New Member
Jan 11, 2021
15
1
3
I went through some of the tools in the mellanox folder and found the subnet and started that on both machines. I now have a subnet connection at 40 gbs. It does show as a unidentified network with no network access for both ipv4 and ipv6. They seem to be receiving and sending traffic so it looks like I have a connection between the win10 and server 2019. Will try and do some testing to verify. Thanks again for the help.
 

datanet

New Member
Apr 29, 2020
6
4
3
I used this when I needed to understand:


I have some ConnectX-3's as well. Whether you connect NIC to NIC or NIC to switch, you want a B cable that reverses the fibers. You should only need 8 fibers (4 send at 10Gb per fiber and 4 receive at 10Gb per fiber), but I bought 12 fiber cables off eBay since the ones I found were cheap (personal seller trying to offload extras). The 12 fiber cables work fine. I bought some Brocade 40Gb QSFP's (search for Brocade 57-1000267-01 XBR-000232) and use female B to female B. Hope this helps.
The part number you listed for the transceiver (Brocade 57-1000267-01 XBR-000232) is for fiber channel, but you also referenced 40GbE. I just want to confirm you listed the right part number since i doubt that transceiver will support 40GbE connections.
 

MoMeanMugs

Member
Apr 16, 2018
32
3
8
71
The part number you listed for the transceiver (Brocade 57-1000267-01 XBR-000232) is for fiber channel, but you also referenced 40GbE. I just want to confirm you listed the right part number since i doubt that transceiver will support 40GbE connections.
It's just a dumb transceiver that's not protocol specific. It has 4 lanes of 10+ Gbps. It works fine for 40GbE. It's the right part number. I use 4 for connections to TrueNAS and 2 as trunk cables between ICX6650's. Everything works as expected.
 

datanet

New Member
Apr 29, 2020
6
4
3
That is really interesting. According to the doc here https://docs.broadcom.com/doc/12379862 it shows it specifically as a fiber channel transceiver. If you're saying you have these exact transceivers and they are linking at 40GbE between the ConnectX-3 and the 6650, then I will pull the trigger and purchase these to resolve the issues with using Mellanox DAC cables between the ConnectX-3 and the 6650 and not having all the ports working.
 

datanet

New Member
Apr 29, 2020
6
4
3
@MoMeanMugs you won't have to be eating the transceivers. ;)

I can confirm those optical transceivers work perfectly fine in the ICX6650 and in the Mellanox ConnectX-3 adapters. Everything links up on any of the 4 rear ports of the ICX6650 and all links show as 40GbE connection.

For easy reference to anyone having this issue in the future, as was already mentioned above, just search for "Brocade 57-1000267-01 XBR-000232" on eBay. I paid a little under $10ea for these. The MPO fiber cable I used was this one from Amazon

The important thing to search for with the fiber patch cable is MPO to MPO female connector, type B polarity. You can use 8, 12, or 24 (whichever you can find cheaper) since the transceiver only uses 8 fiber strands (4 send/4 receive - 4X10GbE).

All in, you're probably right around $50 per connection...but it works without any issues.
 
  • Like
Reactions: klui

klui

Active Member
Feb 3, 2019
291
123
43
@datanet, I just want to thank you for this tip. These transceivers all work in the following switches, although I've only confirmed link--also with a CX3.

Brocade ICX6610
Celestica D4040
Mellanox SX6012
VDX 6740 (sidenote, only supports active DACs)
 

digity

Member
Jun 3, 2017
41
1
8
51
Does the transceiver and MPO cables mentioned in this post make the QSFP+ (40 Gbe) to 4 x SFP+ (10 Gbe) breakout ports behave like regular 40 Gbe port on the ICX6610 and ICX6650 models? Or does this transceiver & MPO cable combo only work on the normal 40 Gbe ports?
 

klui

Active Member
Feb 3, 2019
291
123
43
Does the transceiver and MPO cables mentioned in this post make the QSFP+ (40 Gbe) to 4 x SFP+ (10 Gbe) breakout ports behave like regular 40 Gbe port on the ICX6610 and ICX6650 models? Or does this transceiver & MPO cable combo only work on the normal 40 Gbe ports?
Only 40G ports. They don't link up with the cable. The switch still displays "error in reading qsfp" if it's inserted in the 40G ports but link up. You need a breakout or cassette w/ breakouts.

ICX6610-48P Router#M:9 L:0 - chow_qsfp_read, qsfp 2, error in reading qsfp
M:9 L:0 - link_40G_4x10G_get_media: qsfp 2, port 1/2/2 error in reading qsfp
------------------------------------------------------------------
------------------------------------------------------------------
chow_40G_4x10G_get_media: error in reading qsfp 1/2/2
ICX6610-48P Router#show media
Port 1/2/1: Type : EMPTY
Port 1/2/2: Type : 40GBASE-Passive Copper
Port 1/2/3: Type : 40GBASE-Passive Copper
Port 1/2/4: Type : 40GBASE-Passive Copper
Port 1/2/5: Type : 40GBASE-Passive Copper
Port 1/2/6: Type : EMPTY
Port 1/2/7: Type : EMPTY
Port 1/2/8: Type : EMPTY
Port 1/2/9: Type : EMPTY
Port 1/2/10: Type : EMPTY
ICX6610-48P Router#M:9 L:0 - chow_qsfp_read, qsfp 0, error in reading qsfp
M:9 L:0 - link_40G_4x10G_get_media: qsfp 0, port 1/2/1 error in reading qsfp
------------------------------------------------------------------
------------------------------------------------------------------

ICX6610-48P Router#sh media
Port 1/2/1: Type : 40GBASE-Passive Copper
Port 1/2/2: Type : EMPTY
Port 1/2/3: Type : EMPTY
Port 1/2/4: Type : EMPTY
Port 1/2/5: Type : EMPTY
Port 1/2/6: Type : EMPTY
Port 1/2/7: Type : EMPTY
Port 1/2/8: Type : EMPTY
Port 1/2/9: Type : EMPTY
Port 1/2/10: Type : EMPTY