I have been running Mellanox QDR Infiniband primarily due to the magnitude of difference in latency using RDMA and its use via SRIOV in containers and VMs. Unfortunately Infiniband in the industry is almost entirely proprietary and PCIE fabrics have been a promising future for years now without much materializing besides much cheaper ICs and their prohibitively expensive dev kits.
I am looking for hardware/software collaborators interested in furthering the development of PCIE fabrics based on IDT retimer and Microsemi Switchtec fabric switch ICs, as well as open softcore FPGA/ASIC IP (Altera is my current platform of choice), RDMA+IP capable linux drivers, and any necessary Ceph RDMA enhancements. The primary hardware target will be for an inexpensive (sub $50) SBC consisting of a single 4-lane PCIE 3.0 socket for attaching a single NVME device, or possibly 1 SATA 3.0 connection, with (likely) an ARM SoC running a Ceph image for a single osd that is accessible over a PCIE or Ethernet fabric. Due to minimal effort the device would likely alternatively provide 1G ethernet that could be used for RoCE unless a cost effective 10G IC is practically discovered.
What this would ultimately look like is a device you could attach a single NVME or SATA device to and have a completely self-contained Ceph OSD accessible by less propriety PCIE or Ethernet fabrics with or without RDMA, with a target of equal or better latency and performance compared to using Infiniband with RDMA. The intent is to reduce the cost and complexity and increase the performance of storage clusters. Future developments could include IP over PCIE.
I am looking for hardware/software collaborators interested in furthering the development of PCIE fabrics based on IDT retimer and Microsemi Switchtec fabric switch ICs, as well as open softcore FPGA/ASIC IP (Altera is my current platform of choice), RDMA+IP capable linux drivers, and any necessary Ceph RDMA enhancements. The primary hardware target will be for an inexpensive (sub $50) SBC consisting of a single 4-lane PCIE 3.0 socket for attaching a single NVME device, or possibly 1 SATA 3.0 connection, with (likely) an ARM SoC running a Ceph image for a single osd that is accessible over a PCIE or Ethernet fabric. Due to minimal effort the device would likely alternatively provide 1G ethernet that could be used for RoCE unless a cost effective 10G IC is practically discovered.
What this would ultimately look like is a device you could attach a single NVME or SATA device to and have a completely self-contained Ceph OSD accessible by less propriety PCIE or Ethernet fabrics with or without RDMA, with a target of equal or better latency and performance compared to using Infiniband with RDMA. The intent is to reduce the cost and complexity and increase the performance of storage clusters. Future developments could include IP over PCIE.
