How many nodes do you have? I've been gauging if I should do 40gb ib over ethernet for a smaller install with 4 nodes.
If you had 4-6 nodes then it's like $600 for 4 cards and $1000 for a switch so $1600 but you usually get 48x 10gb ports as well.
The ib switch for $200 + $25 * 4 for the cards is maybe $300.
You'd also need cables so that would add another $100-150 for each.
I've been struggling with eating the $1000 and getting 10gb ethernet ports as well. Not having to learn ib is good and what I'm more worried about is that I can turn around and plug the ethernet switch into a future network easily.
I'd like to hear how this project turns out.