I made a post over on Reddit about my new cluster setup, but figured I'd post it here as well, since it is very specific to this thread.
Finally got around to setting up my new cluster. My current one is 3 m720q running Proxmox and using their local storage or connected via NFS to my UNRAID server. Suffice to say, the bandwidth performance isn't great. The 1g link is heavily saturated, so doing VM migrations or even booting VMs over NFS is pretty painful.
Ordered a Mellanox SX6036 on eBay for $100 shipped and a couple of Connect X3 cards for about $15 shipped each. So, for less than $200 (with the QSFP cables) I have a 40gbe network going.
The hosts are m720q, 32GB of RAM, a 2TB Intel NVMe on the bottom NVMe port, a 256GB Hynix in the WiFi M.2 slot (for the OS), Connect X3 cards, a 3D printable fan shroud I designed to cool off the cards, and a baffle I designed to allow a clean look on the back side of the server with a notch to allow a USB cables to pass through the back.
Fan Shroud:
www.printables.com
Bracket:
www.printables.com
Running CEPH on the Proxmox cluster with a "RAID5" erasure coded pool, I was able to achieve some pretty impressive speeds!
Pretty dang fast for using storage over the network
This is with only one port used, I'm not sure if these little units can really benefit from using the second port in a LAG setup, but I'm willing to try!
Not sure if I can get that 32gbps up to 40, but it'd be nice (9000 mtu)
With the erasure coded pool, I get about 4TB of usable space, redundancy, and incredible speed. The nice thing is, the more nodes I add to the cluster, the faster the storage gets! Now I wont have to wait minutes for a VM to start up, or worse, a migration taking a LONG LONG time, since all the nodes already have the needed data, it's practically instant.
Just wanted to share!
Power Usage:
Single Node at Idle with a Windows VM = 17w
Single Node with a maxed out network test = 56w
Single Node with a maxed out CPU (No network) = 50w
Switch at idle = 46w
Switch with 100gbps blasting through it... 46w. Lol.