Weighing in as H8Roads Mentioned... We got those chassis, Mine costed less because i had all the gear from other Chassis C6100 and other Whiteboxes that were using same Gen Tech just wanted to consolidate it.... (all i needed was the chassis)
The onboard 10Gb should work without the dongle the dongle is for the use of the Infiniband Side (QSFP Cabling) the SFP for the 10Gb should work without the dongle...
As H8Roads Mentioned there are some catches with ESXi, it doesnt play nice with the Ofed 2.3.3 driver although it does identify both it doesnt play nice inside the logs, its recommended to use one or the other... Ethernet (SFP) or Infiniband (QSFP) but not both at the same time.
FYI as well the Infniband side (QSFP) is not Ethernet capable from what i found out.... It does only infiniband... I use the Ofed 1.8.2.4 With the firmware that we got from HP i think its 2.9.1530 the initial installed was a 2.8 Firmware which would have had a bad time... these are considered ConnectX-2 Based cards - MT26438 - so luckily for me that means Solaris (OmniOS) Supports this natively iir!
The IB switches we both got are in the same generation but we also got a good hookup on them, the OpenSM side will work great for the IB infrastructure... Ethernet side i think u can use pretty much any SFP based switch...
Currently running ESXi 6.0U1 with OFED 1.8.2.4 and all is stable right now at 40Gb Running backended vMotion, vSAN, NFS, etc. for main storage...
Ill have pics later