Here is what I've learned (so far) into my 3mo long 100gbe journey..
You can flash Connectx-5 NICs to be other Connectx-5 versions.. like the dual port 50G to dual port 100G - or a CAT to a DAT and so on. The PCIe 3 x16 cards become the PCIe 4 x16 cards too.. you just have to 1. be willing to risk it and 2. research.
It's a good way to save some $.
If you get used - slap that new-to-you card under a microscope or strong magnifying lens and look for missing (tiny) capacitors and what not. For some reason every (7) used Mellanox NIC I buy has missing caps.
https://www.serversupply.com has good deals on new cards.
Dell cards run fine in most setups and their part numbers sell for less - it's the same card though. HP/Lenovo I would stay away from - for firmware feistiness reasons. Cisco/Supermicro seem fine, but I have no experience with them.
Marvell and Intel make 100G NICs too.. Marvell being just fine and less expensive, Intel has good system compatibly, can run cooler than Nvidia (Mellanox) and costs the most. Intel is also module picky (Intel <3 Intel).
Mellanox is super configurable and runs with any cable or module I throw at it. Nvidia bought them and thankfully - so far - has good docs and driver/FW support. Mellanox cards can run hot, depending on the model. Check the Air Flow specs listed in the docs. Some Connectx-5 run cooler than some Connectx-4. If you don't actually plan to use the specific Connectx-5 features that Connectx-4 doesn't have (not many) then save the coin. Connectx-4 is a solid value.
Skip the made in China/India Mellanox cards, look for Israel.
If you don't get the right height for your adapter be prepared to spend $20 - 30 more or wait for shipping from China at $6 for 6. It's 2 screws and 30 seconds to make the swap.
Mikrotik has the "cheap" new 100G switches - see the reviews from
Mr. Kumar on this site. I blast my Ceph traffic over two of the CRS504 switches and it's snazzy. The Mikrotik learning curve takes some time to suffer through. They don't do
RoCE - so that might be an issue (read: need fast CPU, PCH, etc.).
100G QSFP28 cables and modules cost more.
U.2 is new your friend - she also costs more. Provantage and ACME servers have okay deals.
You'll not stop once you see how awesome 100G is and start upgrading everything else.
Ebay is more enemy than friend, but prepare to spend hours every day making sure they don't list something, you're not sure you even want.
You will be in a much smaller group of homelab bros - expect less help, hand holding, known documented solutions, etc.. The SMB/Enterprise crowd has already moved on and don't have time to help you either. In fact they scoff at your lowly arrogance and pitiful budget. How dare you reach out! Eww!
You will look with disdain upon those plebeians stuck in a 1G/2.5G/10G world, they will hate you for it, and may plot their revenge. To throw them off your scent, embrace "quite luxury" - no blingy RGB LEDs.
In the back dark corners of tech pubs and forums are your people, you'll recognise them by the beards they forgot to shave (even the women).
