That was a quick test using Windows. No other setting changes required. Not using RoCE but no doubt will work. I am not using VMware for RoCE at this time as my fiber buildout is not complete. Honestly I am taking it a step at a time. Get regular ethernet working then think about what stacks to use to get more performance.
@Rand__ is correct. One needs to make sure all components work. CX3s are on the outs with newer VMware versions and I am migrating away from that to Proxmox.
Was my first go-around with any Mellanox stuff, I hadn't thought about testing before modding. In retrospect, sounds like a very good idea. They were all 10GbE/40GbIB cards, so something about the mod must have gone right. Just looked up my purchase history, yep, as I suspected, I messed up and bought QDR cables. Bah - Title:
New Mellanox HP 498386-B24 5M 4x DDR/QDR QSFP IB Fiber Optic Cable (H1290) $25/ea
I can't remember the exact state of mind I was in when I made that blunder, but my guess is something like I wasn't going to pay $50+/cable when the CX3s were $30-$35. But it's kind of nice to know it's not a problem with cards, and I can relax and stop worrying about it.
Re: Linux, it does make more sense to go FOSS for both experimental reasons, or being so tight your ass cheeks could mint a diamond. Why Proxmox instead of just a vanilla distro? I never quite got that, it's just Linux with a web interface.
If you like that kinda stuff, have you seen this new project Harvester? It's another all-in-one solution like Proxmox, but is aimed at converting the paradigm of using VMs "manually" into managing their lifecycles more like microservices:
Harvester GitHub - harvester/harvester: Open source hyperconverged infrastructure (HCI) software
Closed-source stuff is generally more stable, but as a result is always playing feature catch-up - this new kubernetes lifecycle management is some groundbreaking shit. Also, the distributed persistent object storage systems like ceph (or even just block like longhorn and openebs). Pretty mindblowing when it comes down to it.
Kind of worth it just to sit back and reflect for a second on how truly amazing all this stuff is. Makes me feel less bad about buying QDR cables by accident.