Help with Connectx-3 back-to-back using FS QSFP-BD-40G

mattaw

Member
Jul 30, 2018
56
17
8
Hey all,

I used the instructions to cross flash two HP Connextx-3 cards QSFP to Mellanox MCX354A-FCBT's.

I then installed two Fiberstore (FS) QSFP-BD-40G modules into one card configured as dual Ethernet.

No dice, the link won't come up and the lights on the card stay off.

Things I have tried:
1) Different OM3 LC-LC fibers, brand new
2) Swapping order of transceivers in card
3) IB and Ethernet options
4) Long fiber (20M) and short fiber (2M)
5) Firmware updated to latest 2.42.5000
6) Another cross-flashed card
7) Read all the docs I can find
8) Using phone camera to see laser flashes (one fiber pulses about every 0.5s)
9) Reversing the fiber order in the LC-LC cable

It seems on windows you cannot interrogate the transceiver, sadly. I will try on Linux over the weekend.

Any thoughts? Turning auto negotiation off and forcing a speed (doesn't seem possible in windows?). Purchase a cheap DAC cable to test without transceivers?
 

i386

Well-Known Member
Mar 18, 2016
3,389
1,137
113
33
Germany
Can you set the nics to infiniband mode, run the command "ibdiagnet --get_cable_info" and psot the result?
 

mattaw

Member
Jul 30, 2018
56
17
8
I forgot to say I tried that.

On linux ethtool -m works and prints out the transciever info. Serial numbers match labels on the modules.

On Windows mst add cable refuses to add anything and then ibdiagnet complains I have no cables.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,587
2,776
113
31
fohdeesha.com
has anyone confirmed connect x3's will run BiDi optics? I know they take about 3x the power from the QSFP interface as regular SR4 optics, and identify themselves as such (higher power class). I know the SX series mellanox switches will not bring them up unless you compile the firmware with some "high power optics" flag to allow them. Not sure if a Connect x3 has the juice to bring one up, let alone 2 at the same time. Then again, maybe there's a similar flag for mellanox NIC's to allow them

@mb300sd
 

istamov

New Member
Jul 31, 2015
15
5
3
has anyone confirmed connect x3's will run BiDi optics? I know they take about 3x the power from the QSFP interface as regular SR4 optics, and identify themselves as such (higher power class). I know the SX series mellanox switches will not bring them up unless you compile the firmware with some "high power optics" flag to allow them. Not sure if a Connect x3 has the juice to bring one up, let alone 2 at the same time. Then again, maybe there's a similar flag for mellanox NIC's to allow them
According to this post on Mellanox forum, only ConnectX-4 (and above?) supports BiDi optics, as ConnectX-3 provides a maximum of 1.5 Watts.
The CX-4 release notes says that "QSFP-40G-SR-BD (Cisco 40GBASE-SR-BiDi, duplex MMF)" is supported, but no mention for single mode optics. Such info is missing on the CX-3 notes.

I know this is not very useful, at $dayjob we have Cisco QSFP-40G-SR-BD optics, but all are in production and I doubt they will allow me to bring one to the home lab just for testing purposes.
 

mattaw

Member
Jul 30, 2018
56
17
8
Thanks for the info so far. I have borrowed 56 gbps infiniband qsfp cable from a cluster at work and will test tonight.

As for the bidi optics I'm sure I saw Cisco bd as supported in the firmware release notes for 2.52.5000? But maybe two isa bridge too far...
 

mattaw

Member
Jul 30, 2018
56
17
8
Dang, found the 1.5W limit in the specs. All the lc-lc options are 3.5W.

Mpo 20m is at least $300 for cable. Om3 lclc I have run already.

Thoughts anyone? I have a 2m patch to a 6 Port keystone Jack with four filled (one a lclc coupler) into the basement and then an 18m run to the server.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,587
2,776
113
31
fohdeesha.com
300 for one cable!?!? Where are you looking? I have bought plenty of 30m MPO off of ebay for ~$70, sometimes closer to 40
 

mattaw

Member
Jul 30, 2018
56
17
8
Good point, my initial searches showed much more expensive cables than that. I now need to consider Connectx-4 or cabling...
 

mattaw

Member
Jul 30, 2018
56
17
8
I know this is not very useful, at $dayjob we have Cisco QSFP-40G-SR-BD optics, but all are in production and I doubt they will allow me to bring one to the home lab just for testing purposes.
I just checked and both the Connectx-3 and 3-pro firmware release notes list the "40GbE QSFP-40G-SR-BD Cisco 40G BD Module" as tested good. I just need to get one to confirm it, however my Fiberstore BiDi QSFP's clearly don't work. So it is a risk...
 

mattaw

Member
Jul 30, 2018
56
17
8
Can someone post an ethtool <connectx-3> in this thread, along with firmware version and actual card brand? It looks like the connectx-3 had 40000baseKR4 at some point and it was taken away, or the crossflashed cards don't get it.
 

mattaw

Member
Jul 30, 2018
56
17
8
Foolishly still trying to keep the lc-lc dream alive purchased two LR4L's. Will try mmf first (nothing to lose and it's only 20m). After that SMF is very cheap.

Thanks for all the help so far. Now if only I could convince myself to upgrade my Dell switch to an icx6610.
 

i386

Well-Known Member
Mar 18, 2016
3,389
1,137
113
33
Germany
You also need to confirm whether FS has compatible BD for the HP Connextx-3 cards, as HP makes it difficult to code compatible optics.
I tried (original) cisco transceivers and they worked with the hp mellanox cards :D
 

mattaw

Member
Jul 30, 2018
56
17
8
Thanks for taking an interest. A pair of lr4r' s worked great.

My thinking is that one of the fs bidis is broken, as it was direct connect of two mellanox cards.

They were purchased second hand from eBay so it isn't FS - everything I have read suggests they are a great company.

I was able to read their status but they always showed 0 for received laser power.
 

azev

Active Member
Jan 18, 2013
753
236
43
I tried (original) cisco transceivers and they worked with the hp mellanox cards :D
Which Cisco optics did you try this with ?? is it the QSFP-40G-SR-BD ?? I am planning to get a few of these for my storage server and I am using nexus switch.
 

azev

Active Member
Jan 18, 2013
753
236
43
anyone tried using cisco brand QSFP-H40G-CU (Cisco QSFP to QSFP copper direct-attach cables) to connect the nics to cisco nexus switches ?