Well, I got my 10GbE network going. My configuration:
- MikroTik CRS309-1G-8S+IN 10GbE switch (this is the 8 port SFP+ version without fans)
- 3 Mellanox MCX311A-XCAT 10GbE NICs, which I added to:
- HPE MicroServer Gen10+. NIC is on x16 slot.
- 1st HPE ML30 Gen 9 server. The NIC is on a PCI-e port that's x8 physical and x4 electrical
- 2nd HPE ML30 Gen 9 server. The NIC is on a PCI-e port that's x4 physical but only x1 electrical. Finding a 10GbE NIC that was "ideal" for an x1 slot was the challenge (thanks for suggestion to not even consider a PCI-e 2.0 card and just get PCI-e 3.0 card).
I tested network speeds using iperf3 (nice that it's already included as part of ESXi in /usr/lib/vmware/vsan/ bin).
As expected, iperf testing between the MicroServer Gen10+ and the 1st HPE ML30 Gen9 (with NIC in an x4 electrical slot), resulted in 9.27 Gbits/sec.
I was expecting to get less speed with the 2nd HPE ML30 Gen 9, since the NIC is only on an x1 electrical socket. iperf testing between the HPE MicroServer Gen10+ and the 2nd HPE ML30 (which has the NIC in an x1 electrical slot) resulted in 6.11 Gbits/sec. I was hoping something closer to the theoretical 8Gbits/sec of a PCI-e 3.0 x1 slot, but result was still nice compared to what I was getting out of the 1GbE ports. Normally, I shouldn't need to come close to the 6.11 Gbits/sec on my testing. This 2nd HPE ML30 Gen9's x4 and x16 slots are already being used by an NVMe and a GPU, respectively, so an x1 electrical (x4 physical) slot was all that was available for the NIC.
Aside: I also tested iperf3 with my old MicroServer Gen8. It only got 0.55 Gbits/sec, which is a bit lower than the theoretical 1Gbits/sec.
Anyway, the Mellanox NICs were a fine choice. 10Gbe and supported by VMware drivers in ESXi 6.7 and 7.0. SOIV would've been ideal, but the used prices of the Mellanox NICs were enticing enough (< $40/NIC).
Next step is to migrate an old Synology to a newer one with 10GbE NIC, to add on to the network (ultimately, simply transferring the disks from the old NAS to a new NAS is simplest/fastest approach).
Piglover - Thanks for suggestion on SFP+ and cables. Better prices for SFP+ and I suppose better stability and lower energy. SFP+ amongst the equipment in my basement was a good choice. Longer term, I'm still looking for a 10GBASE-T (RJ45 ports) to use in my main equipment closet to reuse the existing CAT 6 wiring throughout my home. Ideally, I'd get one with enough 1GbE and 10GbE ports to replace my main 1GbE Cisco SG-200 switch that's past EOL (I need two to three dozen ports), or just get a 10GBASE-T switch with 8 10GbE ports in the shorter term to supplement my main switch. The MikroTik CRS312-4C+8XG-RM switch (with 8 1G/2.5G/5G/10G Ethernet ports) would have been nice, but as mentioned, the fan is way too loud. The switch would be in an equipment closet, which is not too far from a bedroom, and from the Youtube video's I've seen it sounds obnoxiously loud. If I can find a switch for just a slightly higher cost, it would be worth the extra $$ to not mess with fans, voiding the warranty, etc.
www.vmware.com
.