Mellanox Card choice (VPI or Eth) : need advice

[0xbit]

I'm Trying to decide what Mellanox card to get for a home lab.
MCX354A-FCBT
http://www.mellanox.com/related-docs/prod_adapter_cards/ConnectX3_VPI_Card.pdf
or
MCX314A
http://www.mellanox.com/related-docs/prod_adapter_cards/ConnectX3_EN_Card.pdf

One is a VPI card that supports Infiniband and the other isn't.
As both cards are the same price, I am at a loss for which is better
and what are the tradeoffs if I go for one over the other.

Beyond typical network traffic, I will be experimenting with RDMA over Converged Ethernet (RoCE) and will likely not configure it infiband mode but may down the road w/ another Connectx-3 card purchase. Are there extra software/driver headaches I will have to deal w/ regarding the VPI cards? Anything I should be aware of w.r.t to VPI cards vs. standard ethernet? Any reason to pick one over the other?

I will be using these under Ubuntu linux (16.04+).
Also, I will be using a QSFP+ (40GbE -> 4x10GbE SFP) cable to connect one of the ports to two computers running Connectx-2 10GbE standard cards.

Apologies for being a complete noob
Also, will I run into any issues with this cheap sucker :
* NEW * Volex 2M Passive 40Gb QSFP+ to 4x SFP+ Breakout Cable - VAHS-30-0428-2M | eBay

 

[0xbit]

Also, down the road, is it possible to use a 40GbE QSFP+ DAC cable to directly connect two computers that have such ports?

 

[i386]

VPI Card

Pro:
- Supports Infiniband and Ethernet, great if you want to try infiniband and still have a "usefull" card (because of 40gbe networking).
Con:
- Autodetect doesn't work properly with the mellanox drivers, sometimes ports are configured as ib or ethernet

EN Card

Pro:
- Ethernet only, no auto configuring fails
- (Work in some nas devices with pcie slots)
Con:
- No infiniband

Also, down the road, is it possible to use a 40GbE QSFP+ DAC cable to directly connect two computers that have such ports?

Yes.

Also, will I run into any issues with this cheap sucker

I think that was answered in the other thread, but anyway: It won't work with the mellanox nic qsfp ports. These cables are for switches that can configure a qsfp to 4 sfp+ ports/connectors.

 

[0xbit]

Pro:
- Supports Infiniband and Ethernet, great if you want to try infiniband and still have a "usefull" card (because of 40gbe networking).
Con:
- Autodetect doesn't work properly with the mellanox drivers, sometimes ports are configured as ib or ethernet

Pro:
- Ethernet only, no auto configuring fails
- (Work in some nas devices with pcie slots)
Con:
- No infiniband

Yes.

I think that was answered in the other thread, but anyway: It won't work with the mellanox nic qsfp ports. These cables are for switches that can configure a qsfp to 4 sfp+ ports/connectors.

Thank you so very much for the quick reply !
This helped me finalized my choice. I will go w/ the dual port Connectx-3 VPI QSFP+ 40/56 Card :
http://www.mellanox.com/related-docs/prod_adapter_cards/ConnectX3_VPI_Card.pdf
and slap on a QSFP+ to SFP+ adapter to be compatible with and connect to my other computer hosting a Connectx-2 EN 10GbE card.

An extra $20 for the adapter and I get a card capable of 40/56 vs. 10/10 and also get the ability to play with IB down the road.
I think I will have enough goodies to keep me occupied. Thank you so very much for this info. Although I had researched this for some time, I was getting hung up on the [buy].

This community is really comfy here ^_^

 

[Biren78]

I like VPI cards or Ethernet better than pure IB. There are few times I've ever thought about saying F* Ethernet and going IB. But I've had the opposite thought for network segments. If you're getting VPI or pure Ethernet you're not going to have issues changing to Ethernet for a direct connect.

 

[azev]

has anyone experienced connecting these vpi card to nexus switches ?? I am very interested in this and would like to know if this would work connected to nexus 40gb qsfp port ??? if so what optics do you use ?? will the cisco bi-di 40gb optic works ?? how about qsfp copper cable ?? should I get cisco branded one or mellanox for compatibility purposes ?