I need an infiniband cable and I'm lost

[Oni.kage]

I just ordered two ConnectX-2 cards MHQH19B-XTR with the intention of doing a point-to-point connection from my NAS (Ubuntu/ZFS) to my Windows desktop. However, I have no idea what kind of cable I should buy or how much I should spend on it. Could I have some help picking one out?

I need a cable that's 25-30 feet long (8-10 meters).
Since I'm doing point-to-point I'll need dual QSFP connectors?
Fiber or copper? I really need some advice here. Fiber is expensive, but I might be willing to do it if the latency will be that much better.
Does the cable have to be Mellanox brand, or will something like this be fine?

I've never worked with infiniband before. I know the X-2s are capable of 40gbps, but I don't think I can utilize that speed using IPoIB? How will my choice in cable affect latency/max bandwidth? What is the best bang-for-buck for my needs?

Appreciate the help.

 

[dba]

You need a cable that's QSFP on both ends and rated to handle QDR Infiniband at 40Gbits/s. On eBay, you might find one for $30 to $100 depending on your luck. Do an eBay search for "QSFP cable" and sort by cost. In your hunting, give preference to Mellanox cables, if only because they are a known entity. Here's an example of a 10 meter Mellanox active copper cable from a US vendor at a price that is very reasonable - $72 including shipping:

Mellanox MCC4R28C 010 33 Feet Infiniband Cable 1 x QSFP Connector | eBay

With some patience you may even find one for less. You sometimes see extremely long $1,000 fiber cables going for $30.

I just ordered two ConnectX-2 cards MHQH19B-XTR with the intention of doing a point-to-point connection from my NAS (Ubuntu/ZFS) to my Windows desktop. However, I have no idea what kind of cable I should buy or how much I should spend on it. Could I have some help picking one out?

I need a cable that's 25-30 feet long (8-10 meters).
Since I'm doing point-to-point I'll need dual QSFP connectors?
Fiber or copper? I really need some advice here. Fiber is expensive, but I might be willing to do it if the latency will be that much better.
Does the cable have to be Mellanox brand, or will something like this be fine?

I've never worked with infiniband before. I know the X-2s are capable of 40gbps, but I don't think I can utilize that speed using IPoIB? How will my choice in cable affect latency/max bandwidth? What is the best bang-for-buck for my needs?

Appreciate the help.

 

[33_viper_33]

Those are the VPI adapters which use the QSFP connectors. If you do a ebay search for “infiniband qsfp” you will find what you are looking for. Here are two examples:
Mellanox MCD4Q26C 008 26 Feet Infiniband Network Cable QSFP | eBay
Used Mellanox Technologies 30M 40GBS Infiniband QSFP Op Cable | eBay

Fiber vs copper depends on length of run and how noisy of an environment you are in. It sounds like this is for a home setup. If so, you are probably fine with copper unless it is running across you main electrical panel. These copper cables are shielded and you will likely be fine with copper up to that length. If you take a look, you will find that used fiber is not much more expensive if you are patient and willing to accept alot of excess.

As for speed, it really depends on setup and what you are trying to accomplish with it. If you have lots of parallel workloads, Infiniband will shine. If you are talking about a single file transfer, it will be quick but not necessarly as amazing. OS and protocols matters a lot here. Windows 8 and 2012 are capable of RDMA which will utilize all 4 x 10gbps pipes for a single file transfer job. I’m finding windows 7 and below will only use one at a time for a single file transfer job which means it maxes out at roughly 10gbps. I’m currently testing this and my results can be found here:
http://forums.servethehome.com/networking/2781-infiniband-testing-high-speed-low-power-aio.html

 

[Oni.kage]

Thanks for the help gents.

I was pretty satisfied with the price on that optical cable you linked viper and bought it. No other mellanox fiber cables on ebay were close that I could find. I also like the extra slack because I may need to move the server farther away in the future (it's quite incompatible with any creatures that have ears).