The Broadcom PEX88000 PCIE 4 and Broadcom PEX9700 PCIE 3 family of chips have some very intriguing features, most interestingly Tunneled Windows Connection (TWC) that allows true multi-host to host communication via DMA. The previous generation PEX8700 only supported Non-Transparent Bridge (NTB) for more of a host failover cluster scenario. These chips allow SR-IOV Virtual Functions (VFs) from one host to be utilized in a completely different host. To top it off, they support a Virtual Ethernet NIC DMA for host-to-host communication instead of directly using TWC. At PCIE 3 x16 performance would theoretically exceed Infiniband 100Gb EDR by 25Gb/s and PCIE 4 x16 would have x2.5 EDR bandwidth at around 250Gb/s. Latency would be on par or slightly exceed RDMA without the application customization RDMA requires.
All that being said hardware is scarce or unobtanium. I did stumble upon the Broadcom P411W-32P that has the PEX88048 which pc-pitstop and provantage seem to sell new at prices somewhat on par with eBay EDR HBAs. Although it's targeted for NVMe attachment without requiring bi-furcation, I'm wondering if it could be purposed towards multi-host NIC DMA without impossible firmware modification.
All that being said hardware is scarce or unobtanium. I did stumble upon the Broadcom P411W-32P that has the PEX88048 which pc-pitstop and provantage seem to sell new at prices somewhat on par with eBay EDR HBAs. Although it's targeted for NVMe attachment without requiring bi-furcation, I'm wondering if it could be purposed towards multi-host NIC DMA without impossible firmware modification.