Doing switched Infiniband on the cheap?

[Agrikk]

Hey Y'all,

I've recently found this site and it's great to see a bunch of gear junkies doing crazy things with cool kit in their homes!

I've recently become enamored with Infiniband and am looking to replace my 1gig iSCSI network with Infiniband to become familiar with the technology. No, I don't need the speed boost as my ESX cluster and SQL Server 2012 live quite happily with MPIO over a pair of 1gig links. I just think that Infiniband has a coolness factor that I want to explore.

That said, I've made a few mis-purchases for Infiniband HBAs off of ebay that resulted in incompatible and unsupported hardware.

Rather than reinvent the wheel here, can anyone suggest a cheap-as-hell setup that is ESXi and Windows Server 2012R2 compatible?

I'm looking for a managed switch (so I don't have to run a subnet manager on a box) and 5 HBA cards capable of running RDMA. THe cheaper the better.

Can anyone recommend a switch and the make/model of a 1- or 2- port PCI-e card that I can pull from eBay for cheap?

My initial plan would be to build a storage server and get a point to point setup working, then expand it with additional hosts and a switch.

 

[Mymlan]

If you're just looking for an affordable, low-feature 10Gb IB setup, I recommend picking up a used Topspin 120 (or Cisco SFS 7000) and a couple of Mellanox MHEA28-XTC NICs. They're cheap, they work, and they're a great start for IB labs.

 

[Agrikk]

I've seen those cards on ebay and I have them in my watch list. However, are they RDMA capable and do they work with Windows 2012 or 2012 R2?

edit: this page says they're compatible up to 2008R2. One of the things I had in mind was to take advantage of storage tiering available in 2012R2 so I need a more modern card that'll be supported.

So I found This Mellanox page that shows the driver compatibility matrix. It looks like the ConnectX-2 EN card is the lowest I can go for 2012 support, but still no support for 2012R2.

But these are fiber cards. Are there Fiber to CX4 cables available and would they work with a Topspin 120?

 

[Mymlan]

A couple of things to note. First of all, the ConnectX-2 EN cards are Ethernet, not IB. You're looking for the VPI adapters. In addition, the cards are not fiber, they're QSFP which support a number of transceivers. You can purchase direct-attach QSFP-to-CX4 cables from several vendors, but I recommend picking them up from Ebay for cheap. Once converted they'll run native copper and link into the Topspin as you would expect, although their bandwidth will be bottlenecked by the aged switch.

 

[Agrikk]

Thanks for the tip, Mymlan. I'm still working my way through all the IB stuff and anything you point out is a pain point I miss later.