Infiniband - help on what to buy

[brianmat]

I spent the better part of last night going through the infiniband threads here and while I understand the concepts and processes, I am trying to make some heads or tails out of specifically what I would look to purchase in my environment.

The current environment is as follows:

1. Supermicro 36 bay server chassis with an X9DR3-LN4F+-B running as an All-In-One ESX/OI+Napp-IT ZFS server.
2. Dell R710 ESX server
3. Supermicro SC846TQ (Supermicro 24 Bay SATA 4U AMD QC 1 8GHz 8GB H8DME 2 SIM1U SC846TQ Server | eBay) standalone OmniOS/Napp-It ZFS server. This is for storing backups and our PHD Virtual appliances but not running VM storage.

My goals for this year:

Sell off our MD3000i/MD1000 SAN and use the cash to purchase a Dell C6100 to combine the 3 ESX servers. Convert #1 into a pure OmniOS/Napp-IT ZFS server. Keep #3 as is with the exception of re-purposing the 192GB to the C6100 and adding new RAM to #1.

I am looking to direct connect for the moment between #1, #2, and #3 due to the cost of switches. A long-term plan is to integrate a true switch, but we're trying to manage the costs as we go.

With that being said, can I virtualize a subnet manager server or would it have to be a physical server?

Our storage is NFS to ESX and no iSCSI going forward.

I would love to just be able to throw some Mellanox InfiniHost MHGA28-XTC cards for $40 into the servers just to get some high speed bandwith out there, but I'm not sure if our environment and my desires would work with that. That's the reason for the more specific hardware suggestions. Honestly, 10Gb would probably be plenty for us but if going to 40Gb keeps the price within reason ($1000) then I would just rather do that. This is more for vMotion, backups, and for some later plans to do some array replication. If going to a $250 card saves time and headache then that would be money well spent.

I'm not sure if that provides enough detail about the hardware configuration or not. I can provide other information as requested. Any suggestions are appreciated.

 

[33_viper_33]

I have two HP versions of the Mallanox Connect X cards I'm getting ready to sell since I just purchased newer versions. PM me if you are interested. What specific questions do you have. I'm far from an expert on Infiniband but I am learning. If I can't answer your question, there are many here that probably can.

 

[33_viper_33]

With that being said, can I virtualize a subnet manager server or would it have to be a physical server?

Just took another look at you thread and missed your question. You must have at least one SM on the IB network. OpenSM is available for download for ESXi. If you are running a standalone Windows box, the OFED driver has a SM built into it and can be activated easily. I'm in the middle of another move, but if you give me a couple weeks to get my main computer back online, I'll send you a couple useful links for working with the SM in various OSes.

My experience is IB is more challenging to setup initially, but has the cheep aspect that is advantageous especially if you go with DDR speeds. It also is more future-proof as far as speed is concerned. On the flip side, 10GBE is a bit more expensive for the cards and WAY more expensive for switches. But it is more plug-and-play and uses standard network infrastructure (CAT5E/CAT6). If you are doing connections that are physically close together (such as a rack), I like IB so far. This is how I intend to utilize IB. All my long runs are 10GBE.

 

[brianmat]

This is all within a single rack, so no long runs at all.

Are there any inexpensive eBay switches to look for with SM running there or is that a much more expensive managed option? I am seeing some infiniband switches for under $500, but I am having difficulty determining the capabilities unless I need to be looking for some other keyword to clue me in. 10GBE is just too expensive for the moment based on our needs. I'd have a hard time justifying that cost for anything other than to say we are running 10GBE.

I'll keep an eye out for those links since I need to get some equipment sold before I make my next set of purchases.

If I can just find a good list of OI/OmniOS supported (older) Mellanox cards then I think I will have a good start.

 

[markpower28]

for Mellanox MHQH19B-XTR. This is a good deal, this guy takes offers. I got two for $200. And it will worth the investment for something like a 4036. You can get it around $1,200-$1,500 at the moment

Mellanox MHQH19B XTR PCI E Single Port Connectx 2 40GBPS Network Adapter | eBay

 

[33_viper_33]

These are QDR (Quad Data Rate) cards which is 40Gb/s. I went with this option because cables are more standardized with QSFP. If you want cheaper, you can find SDR (Single Data Rate, 10Gb/s) or DDR (Dual Data Rate, 20Gb/s) switches for $200-$600 and cards for under $150. These options use CX4 connectors though. QDR is backwards compatible with SDR and DDR but conversion cables cost a healthy sum. I don't have any experience with switches yet, but my understanding is they are all managed (someone set the record straight if I'm wrong). This is where the SM normally resides in an IB network

The cards I am about to sell are HP branded Mellanox ConnectX VPI QDR cards with dual QSFP ports. They come with low profile brackets. They may be a good fit for you especially if you don't intend to buy a switch immediately. They will be on Ebay in about 3 weeks and also posted in the WTS catagory. If anyone is interested, I'll give a better price to STH buyers. Send me a PM.

 

[Darkytoo]

This is all within a single rack, so no long runs at all.

Are there any inexpensive eBay switches to look for with SM running there or is that a much more expensive managed option? I am seeing some infiniband switches for under $500, but I am having difficulty determining the capabilities unless I need to be looking for some other keyword to clue me in. 10GBE is just too expensive for the moment based on our needs. I'd have a hard time justifying that cost for anything other than to say we are running 10GBE.

I'll keep an eye out for those links since I need to get some equipment sold before I make my next set of purchases.

If I can just find a good list of OI/OmniOS supported (older) Mellanox cards then I think I will have a good start.

As someone who fell for how great infiniband is, I can offer some advice.
1. If you like to live on the edge (i.e. upgrade software, use cool features, etc) then do not purchase anything less than a Connect-X 2. ConnectX cards are no longer supported by vmware ESX, and you have to use the older driver which only works with older card firmware and doesn't apparently support anything than iSCSI. you also either need a QSFP switch or conversion cables which are very $$$
2. If you're using windows hyper-V, then you can use connectX cards, they are supported, but only for iSCSI and IPoIB. There are supposed to be openfabrics drivers that give you RDMA, but my best guess is openfabrics is dead and they will never update software, so you're once again stuck with iSCSI.
3. Used switches you get little to no help, so research, research, research. I learned the lesson the hardway when I didn't pay close enough attention when I bought a switch and I thought it was DDR, but it was only SDR, and it barely works.

So, if you want to use infiniband how you want to, buy connectx-2 cards, QSFP cables and QSFP switch. Otherwise, you're gonna regret it.

 

[Patrick]

I would certainly echo these comments. ConnectX-2 or newer at this point.

 

[brianmat]

Good information all. Thanks. Still doing my research just so I don't end up rebuying equipment.