I want to share following testing with you
4 PVE Nodes cluster with 3 Ceph Bluestore Node, total of 36 OSD.
Ceph over RDMA - rados bench -p rbd 60 write -b 4M -t 16
2454.72 MB/s
Standard TCP/IP - rados bench -p rbd 60 write -b 4M -t 16
2053.9 MB/s
Total performance gain is about 25%
Total pool performance with 4 tests running - rados bench -p rbd 60 write -b 4M -t 16
4856.72 MB/s
4 PVE Nodes cluster with 3 Ceph Bluestore Node, total of 36 OSD.
- OSD: st6000nm0034
- block.db & block.wal device: Samsung sm961 512GB
- NIC: Mellanox Connectx3 VPI dual port 40 Gbps
- Switch: Mellanox sx6036T
- Network: IPoIB separated public network & cluster network
Ceph over RDMA - rados bench -p rbd 60 write -b 4M -t 16
2454.72 MB/s
Standard TCP/IP - rados bench -p rbd 60 write -b 4M -t 16
2053.9 MB/s
Total performance gain is about 25%
Total pool performance with 4 tests running - rados bench -p rbd 60 write -b 4M -t 16
4856.72 MB/s