Open Hardware for PCIE fabric

Would I be interested if this kind of hardware existed and was affordable

  • No

    Votes: 0 0.0%
  • I have no clue what you are talking about

    Votes: 0 0.0%

  • Total voters
    4

Ethan Waldo

New Member
Dec 9, 2018
10
2
3
I have been running Mellanox QDR Infiniband primarily due to the magnitude of difference in latency using RDMA and its use via SRIOV in containers and VMs. Unfortunately Infiniband in the industry is almost entirely proprietary and PCIE fabrics have been a promising future for years now without much materializing besides much cheaper ICs and their prohibitively expensive dev kits.

I am looking for hardware/software collaborators interested in furthering the development of PCIE fabrics based on IDT retimer and Microsemi Switchtec fabric switch ICs, as well as open softcore FPGA/ASIC IP (Altera is my current platform of choice), RDMA+IP capable linux drivers, and any necessary Ceph RDMA enhancements. The primary hardware target will be for an inexpensive (sub $50) SBC consisting of a single 4-lane PCIE 3.0 socket for attaching a single NVME device, or possibly 1 SATA 3.0 connection, with (likely) an ARM SoC running a Ceph image for a single osd that is accessible over a PCIE or Ethernet fabric. Due to minimal effort the device would likely alternatively provide 1G ethernet that could be used for RoCE unless a cost effective 10G IC is practically discovered.

What this would ultimately look like is a device you could attach a single NVME or SATA device to and have a completely self-contained Ceph OSD accessible by less propriety PCIE or Ethernet fabrics with or without RDMA, with a target of equal or better latency and performance compared to using Infiniband with RDMA. The intent is to reduce the cost and complexity and increase the performance of storage clusters. Future developments could include IP over PCIE.
 

Ethan Waldo

New Member
Dec 9, 2018
10
2
3
CAPI, CCIX, GENZ
https://opencapi.org/
Cache Coherent Interconnect for Accelerators (CCIX) - WikiChip
Gen-Z Consortium: Computer Industry Alliance Revolutionizing Data Access

These are all open pcie fabric standards.
Intel is pushing one very similar to CCIX for pcie 5 but it is not open, though they claim they will open it up on the 2nd version.
Unfortunately standards to affordable silicon let alone devices and software will be a long cycle. Definitely good to keep it mind for the stack. Thank you for pointing that out.