@Lance Joseph: Any chance you got this to work?
I'd tried the Mellanox Connectx-3 cards with the 4-to-1 (QSFP+ -> SFP+) cable but only got link on the first SFP+ cable.
It may be possible to get this to work, but that's not a rabbit hole I'm prepared to go down at the moment.
I was going to pick up a pair of XL710-DA2 cards, but my vendor said there was a hardware issue with the currently available revision.
That may have been fixed since I last inquired but haven't had the time or inclination to follow up just yet.
There's a new testbed that I created with three systems with various flavors of Connectx-3 cards.
I'm doing some tests with Storage Spaces Direct, but once I finish, I'll bridge the adapters in pfSense.
From what I've read, there shouldn't be any major issues in getting these Mellanox cards working on FreeBSD.
I've a pfSense box running a i5-3330s, 4G RAM, and two X520-DA2's.
You might find some mention of it in one of my previous posts on this forum.
Iperf is able to move traffic across the bridge (going client to client) at around 9.5Gb/s.
If I run an iperf server on the pfSense box itself, three clients peg throughput at around 26Gb/s.