Yes CX3 work in ESX, never passed through to FN.Any confirm if this work with ESXi 6.7U1 (and possibly pass through to FreeNAS - I know Chelsio is the FreeNAS "preferred" card)? I don't see it on the HCL.
Told you soBought a ICX 6450-24 today and planned on using the integrated X540-AT2 + Mikrotik S+RJ10, but think 2 of these (1 per ESXi Host) + 2 x DAC may be a better play.
I think RDMA v1 might not have been on CX2 but am not sure.any major advantages for this card to my older ConnectX-2 VPI?
Let me know how t730 copes with them, and whether they work better in SRIOV-land...I still didn't have a chance to install those ConnectX2 IB/40GbE cards I got a while ago.I'm a few weeks ahead on this one, I picked up a couple of these around $30/ea and a couple Sun branded cards for $27-28 for my other machines. Thanks for the heads up though!
They work great with the T730. SR-IOV is officially supported and works fine; I think I set the system fan a notch or two above half to keep the NIC within the operating temp range.Let me know how t730 copes with them, and whether they work better in SRIOV-land...I still didn't have a chance to install those ConnectX2 IB/40GbE cards I got a while ago.
Yes CX3 work in ESX, never passed through to FN.
Not sure it has actual advantages over virtualized NIC managed by ESX
Told you so
But I am not clear what you want to do there. The 6450-24 has SFP+, so you need QSFP to SFP+ converter like
Mellanox MAM1Q00A-QSA Ethernet Cable Adapter 40Gb/s to 10Gb/s QSFP to SFP+
with that you can attach the card to the switch.
You can do a direct box2box connection with the second port though (with QSFP cable).
But witch CX3 you can't pass through half of the card to FN, nor can you pass a SRIOV adapter to FN.
So in your case it would be good to leave the card in ESX, create two (d)vSwitches, one for direct connection (10.10.5.x) and one for the regular ESX traffic (10.10.1.x).
All ESX services that require the fast link (nfs/iScsi) will be mounted on .5
For all other communication you use the .1 net
Direct connection would be at 40GBe if your pools can manage that, else no need o/cBy direct connection you mean?
- [if that is a reference to prior discussion, no need (unless I'm missing something) for a "switch less" connection with a proper switch / adapters in place (soon).
- Don't disagree with subnetting o/c.
- Since you didn't tell me up front I needed a 3rd ESXi host (joking), but that being on my road map, and also building a new workstation (which I leave on 24/7 anyway), would it make sense to "dual-purpose" that build to use as a 3rd host, but only to serve as a witness?
- The only downside I can think of is that I'd take a performance hit vs. baremetal.
In your case I'd suggest to go the simple way first to establish functionality and then experiment with a fall back plan.
Re 'Told you so' - I told you that you you will end up with a switch sooner or later
Re QSFP - There are also SFP+ 10G Cards from MLX, but why if you can get 40G for free (or the cost of an adapter)
Direct connection would be at 40GBe if your pools can manage that, else no need o/c
I didn't mention third host? But most likely I mentioned the advantages of four hosts for vSan didnt I
Workstation as witness... o/c that will work.
However, if you have a beefy box that is always on why not use it for a third host in the first place?
You'll need to run ESX anyway for the witness vm. Although now that I type this, I am not sure. Have never tried running that on Player/Workstation, might actually work.
O/C a third vsan box would need the same storage HW.
er... NetApp 112-00177 X6558-R5 2M SAS QSFP-QSFP External SAS Cable - 10ft. | eBayI got one A2 card and one A4 card from "usedservers-com*". Came with LP brackets, ordered some regular brackets from China...ETA unknown.
Possibly dumb question: Is a QSFP cable a QSFP cable, or are there differences between, say, a "3 GB/s" QSFP cable from a SAN (specifically, EMC Amphenol 038-003-703 85" 2.2M 3GB/s QSFP to QSFP Male to Male Cable Black | eBay) and one from Mellanox like zer0sum linked?