At work, I have four cluster nodes with 25 GbE that I’m about to connect via Mellanox’s host-chaining feature. We have a 10-12 workstations (also with 25 GbE cards) that would be great to use as cluster nodes on occasion.
I’d love to set this switch up on an unroutable subnet for HPC traffic only. A new PFC/ROCE switch seems to start at $3K at least. I bet that with only 16 nodes, zero-touch ROCE would be reasonably efficient.
(If anyone has experience with ZTR, I’m all ears).