Help with Connectx-3 back-to-back using FS QSFP-BD-40G

[mattaw]

Hey all,

I used the instructions to cross flash two HP Connextx-3 cards QSFP to Mellanox MCX354A-FCBT's.

I then installed two Fiberstore (FS) QSFP-BD-40G modules into one card configured as dual Ethernet.

No dice, the link won't come up and the lights on the card stay off.

Things I have tried:
1) Different OM3 LC-LC fibers, brand new
2) Swapping order of transceivers in card
3) IB and Ethernet options
4) Long fiber (20M) and short fiber (2M)
5) Firmware updated to latest 2.42.5000
6) Another cross-flashed card
7) Read all the docs I can find
8) Using phone camera to see laser flashes (one fiber pulses about every 0.5s)
9) Reversing the fiber order in the LC-LC cable

It seems on windows you cannot interrogate the transceiver, sadly. I will try on Linux over the weekend.

Any thoughts? Turning auto negotiation off and forcing a speed (doesn't seem possible in windows?). Purchase a cheap DAC cable to test without transceivers?

 

[i386]

Can you set the nics to infiniband mode, run the command "ibdiagnet --get_cable_info" and psot the result?

 

[mattaw]

I forgot to say I tried that.

On linux ethtool -m works and prints out the transciever info. Serial numbers match labels on the modules.

On Windows mst add cable refuses to add anything and then ibdiagnet complains I have no cables.

 

[fohdeesha]

has anyone confirmed connect x3's will run BiDi optics? I know they take about 3x the power from the QSFP interface as regular SR4 optics, and identify themselves as such (higher power class). I know the SX series mellanox switches will not bring them up unless you compile the firmware with some "high power optics" flag to allow them. Not sure if a Connect x3 has the juice to bring one up, let alone 2 at the same time. Then again, maybe there's a similar flag for mellanox NIC's to allow them

@mb300sd

 

[istamov]

has anyone confirmed connect x3's will run BiDi optics? I know they take about 3x the power from the QSFP interface as regular SR4 optics, and identify themselves as such (higher power class). I know the SX series mellanox switches will not bring them up unless you compile the firmware with some "high power optics" flag to allow them. Not sure if a Connect x3 has the juice to bring one up, let alone 2 at the same time. Then again, maybe there's a similar flag for mellanox NIC's to allow them

According to this post on Mellanox forum, only ConnectX-4 (and above?) supports BiDi optics, as ConnectX-3 provides a maximum of 1.5 Watts.
The CX-4 release notes says that "QSFP-40G-SR-BD (Cisco 40GBASE-SR-BiDi, duplex MMF)" is supported, but no mention for single mode optics. Such info is missing on the CX-3 notes.

I know this is not very useful, at $dayjob we have Cisco QSFP-40G-SR-BD optics, but all are in production and I doubt they will allow me to bring one to the home lab just for testing purposes.

 

[mattaw]

Thanks for the info so far. I have borrowed 56 gbps infiniband qsfp cable from a cluster at work and will test tonight.

As for the bidi optics I'm sure I saw Cisco bd as supported in the firmware release notes for 2.52.5000? But maybe two isa bridge too far...

 

[mattaw]

Dang, found the 1.5W limit in the specs. All the lc-lc options are 3.5W.

Mpo 20m is at least $300 for cable. Om3 lclc I have run already.

Thoughts anyone? I have a 2m patch to a 6 Port keystone Jack with four filled (one a lclc coupler) into the basement and then an 18m run to the server.

 

[fohdeesha]

300 for one cable!?!? Where are you looking? I have bought plenty of 30m MPO off of ebay for ~$70, sometimes closer to 40

 

[mattaw]

Good point, my initial searches showed much more expensive cables than that. I now need to consider Connectx-4 or cabling...

 

[mattaw]

I know this is not very useful, at $dayjob we have Cisco QSFP-40G-SR-BD optics, but all are in production and I doubt they will allow me to bring one to the home lab just for testing purposes.

I just checked and both the Connectx-3 and 3-pro firmware release notes list the "40GbE QSFP-40G-SR-BD Cisco 40G BD Module" as tested good. I just need to get one to confirm it, however my Fiberstore BiDi QSFP's clearly don't work. So it is a risk...

 

[mattaw]

Interesting news folks: http://www.mellanox.com/related-doc..._Dual_QSFP+_Port_Adapter_Card_User_Manual.pdf states (7.6 MCX354A-FCB[T/S] Specifications) that it supports 3.5W.

So those should have worked with very high fan speeds. I'll try cranking up the server fans now!

 

[mattaw]

Can someone post an ethtool <connectx-3> in this thread, along with firmware version and actual card brand? It looks like the connectx-3 had 40000baseKR4 at some point and it was taken away, or the crossflashed cards don't get it.

 

[fohdeesha]

Here you go, $40 and includes transcievers at each end (and yes, it will work)

IBM 59Y1930 30M QLogic Infiniband QDR QSFP Optical Cable | eBay

 

[mattaw]

Foolishly still trying to keep the lc-lc dream alive purchased two LR4L's. Will try mmf first (nothing to lose and it's only 20m). After that SMF is very cheap.

Thanks for all the help so far. Now if only I could convince myself to upgrade my Dell switch to an icx6610.

 

[i386]

You also need to confirm whether FS has compatible BD for the HP Connextx-3 cards, as HP makes it difficult to code compatible optics.

I tried (original) cisco transceivers and they worked with the hp mellanox cards

 

[mattaw]

Thanks for taking an interest. A pair of lr4r' s worked great.

My thinking is that one of the fs bidis is broken, as it was direct connect of two mellanox cards.

They were purchased second hand from eBay so it isn't FS - everything I have read suggests they are a great company.

I was able to read their status but they always showed 0 for received laser power.

 

[azev]

I tried (original) cisco transceivers and they worked with the hp mellanox cards

Which Cisco optics did you try this with ?? is it the QSFP-40G-SR-BD ?? I am planning to get a few of these for my storage server and I am using nexus switch.

 

[i386]

QSFP-40G-SR4:

 

[mattaw]

I ended up with cloned Cisco single mode lr fibers

 

[azev]

anyone tried using cisco brand QSFP-H40G-CU (Cisco QSFP to QSFP copper direct-attach cables) to connect the nics to cisco nexus switches ?