Switch Advice for Mellanox IB QDR networking

[Indecided]

I've got a couple of SuperServer 6027TR-HTQRF nodes sitting around and i've been meaning to put them into a XenServer/ESXi environment with a 2U X9DRi running FreeNAS/Starwinds shared storage.

Now, the 6027TR-HTQRF nodes come stock with a Mellanox Infiniband ConnectX-3 QSFP port. I'm looking at what sort of switching I will need to be able to network up the nodes.

With SFP+ it's quite simple, but the Infiniband stuff is a bit new to me. It would seem that -

1) I would need to be able to put the IB ports into some form of a ethernet mode (VPI?) for it to work as a proper Ethernet device. Will the ports on the 6027TR-HTQRF be able to function in VPI mode?

2) I would need to pick up a Mellanox switch. I don't need anything crazy fancy, it should be an unmanaged model.. but I understand that it needs (*again*) to support VPI mode, which the cheap rebranded EMCs dont. Can anybody recommend a 5+ port switch that would fit this needs, preferally refurb on eBay?
Wold something like a SX6005/SX6012 work (non-EMC, that is)

Based on reading Mellanox documentation it seems only the SX6036G has VPI supported OOTB while others need a paid upgrade key (e.g. SX1036, SX6012..)

3) I would pick up a 1/2 port card for the X9DRi as there is no native ports on board. Should a ConnectX 2 VPI card be sufficient or should I just spring for a Connect-X 3 card?

Appreciate any guidance in this little setup of mine Thanks!

 

[i386]

>Mellanox ConnectX-3 QDR Infiniband 40Gbps Controller
Mellanox stop making infiniband only nics with the connect-x series so the nic in the supermicro server should support vpi, qdr ib + 40 or 10 gbe.
To answer 1) yes it should be possible. (It should also be possible to crossflash the nic with another firmware so that it supports fdr ib and 40/56gbe.)

If you want to use infiniband, you have to buy a mellanox switch.
I would recommend to look for oem branded mellanox switches (especially hp), they are far cheaper than "vanilla" mellanox switches.

I would take the connectx-3 vpi card, they cost a little bit more than the cx-2 but are still cheap (under 40$ per nic).

 

[_alex]

Ethernet or even VPI Switches from Mellanox come at a (very) high premium and often require additional licensing what costs more than the switch alone.

If you can live with the whole fabric not running 'plain ethernet' but ipoib you'd be fine with a dumb infiniband switch and running opensm on the nodes, i.e. IS 5022 for 40G IB.

 

[Indecided]

If that's the case, it may make more sense to simply buy 2x 2 port CX-3 cards, put them into the FreeNAS server and bridge all of them together since they'll be used just for vMotion and run the regular management traffic on the ethernet ports. Anybody done this and have any recommended cards supported by FreeNAS?