Chip InfiniBand cables

[pld]

Hi there
I'm looking for chip InfiniBand cables for ConnectX-3pro adapters. Preferably FDR but QDR is ok. I found the following:
VO-NET 10M 40G QSFP+ AOC 1AT-1Q2M10XX-03A
038-003-699 EMC
038-003-702 EMC
NETAPP 112-00176
Is any of these compatible with InfiniBand? Do SAS QSFP passive cables use the same pinout as InfiniBand?
Thanks in advance for any help
Greg

 

[i386]

I'm using 1m "sas" qsfp cables from netapp for 40GBE networking (actually a 58-0000033-01 from brocade), for FDR/56GBE I used to buy ibm branded mellanox qsfp14 cables (00W0057).

 

[pld]

O, that's nice!
What HCA are you using?


Any other cable recommendations?
Anyone?

 

[Ayrton]

I bought these, and the seller still has some stock (no affiliation)

Mellanox 00W0049 IBM 1m QSFP Passive Copper FDR14 InfiniBand Cable 00W0051 | eBay

ITEM AS PICTURED.

www.ebay.com

 

[pld]

Thanks for the tip. Unfortunately, I live in the EU, thus eBay is not a good option.

 

[pld]

Btw: according to this:

NETAPP 112-00176 - Netapp 0.5M QSFP-QSFP Cable

NetApp X6557 Cable SAS 0.5M

www.kimbrer.com

NetApp 112-00176 is actually NetApp X6557-R6. Which:

40GBASE QSFP+ Passive DAC Cable (QSFP+ to QSFP+)

Tier 1 components, 100% OEM compatible with Amphenol, Arista, Cisco, Dell, Intel, Juniper, Meraki, Mellanox, Molex, 1110401, 1110402, 1110403, Siemon, MSA, generic, data center optics, and OnePort programmable, limited lifetime warranty, free evaluations

approvednetworks.com

is 40Gbase-CR4 and electrical compliant for QDR InfiniBand.

 

[Drewy]

@pld. Where about sin Europe ? I have a number of shortish qdr/40gb cables you can have if you want to pay postage.
some are decent and some are cheap from china, which typically had a 50% failure rate.
haven’t used them for years and most probably never will. Free to a good home.
im in the uk so if your not customs/duty at the moment will be an issue.

 

[pld]

It's very kind. I live Poland. So, shipping takes some effort.

 

[Drewy]

Some effort = almost impossible at the moment. I was in the same position last week trying to get a icx 7250 out of Poland to the uk. Gave up on that

 

[LoneGeek]

I picked up off ebay a lot of 18 Mellanox 00W0049 IBM cables for $100 in 2020.
This is my definition of "cheap".

 

[pld]

I bought QSFP+ AOC 40G 850nm AQOA9N13ADLN0723 cable. According to the specification it is InfiniBand 4x DDR/QDR compliant. Unfortunately after
ibportstate --Direct 0 1 enable
I get: state: Down, and Physical state: PortConfigurationTraining or Disabled alternately. After
ibportstate --Direct 0 1 width 1
(on both ends) i get Physical state: LinkUp but State: Initializing. OpenSM is running on one of the nodes (no switch). And in the OpenSM log:

OpenSM 3.3.23
osm_vendor_init: 1000 pending umads specified
Entering DISCOVERING state
osm_vendor_bind: Mgmt class 0x81 binding to port GUID 0x117500006f916e
osm_vendor_bind: Mgmt class 0x03 binding to port GUID 0x117500006f916e
osm_vendor_bind: Mgmt class 0x04 binding to port GUID 0x117500006f916e
osm_vendor_bind: Mgmt class 0x21 binding to port GUID 0x117500006f916e
osm_opensm_bind: Setting IS_SM on port 0x00117500006f916e
SM port is down
SM port is up
log_send_error: ERR 5411: DR SMP Send completed with error (IB_TIMEOUT) -- dropping
Method 0x1, Attr 0x11, TID 0x12a8
Received SMP on a 1 hop path: Initial path = 0,1, Return path = 0,0
sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT): SubnGet(NodeInfo), attr_mod 0x0, TID 0x12a8
sm_mad_ctrl_send_err_cb: ERR 3120: Timeout while getting attribute 0x11 (NodeInfo); Possible mis-set mkey?
Entering MASTER state

Does it mean cable compatibility problem?

 

[NablaSquaredG]

Hm that's weird.

Have you tried whether Ethernet links?

 

[pld]

I use QLE7340 which are IB only

 

[pld]

Answering to myself

 

[pld]

I bought following SAS cables:
038-003-699 EMC 3GB/S QSFP CABLE 1.73M
NETAPP 112-00176 40cm
Both work stable at QDR/40Gbps. In case of short passive DAC cables it is possible to get much higher bitrate that specified.

 

[pld]

AOC 40G 850nm AQOA9N13ADLN0723
Seems incompatible with QLE7340. Woks only at width 1.

 

[pld]

OpenSM errors were caused by missing software. It seems that QLE7340 needs iwpmd daemon (present in Debian rdma-core package).