I have a small Ceph cluster with 4 nodes, each with 1 2TB spinning disk as an OSD. When I create a block device and run a benchmark like bench.sh, I am only getting around 14MB/s. The raw disk by itself gets somewhere around 85MB/s on the same test, so obviously I am doing something wrong here. I've upped the MTU to 9k on all NICs, though they are just 1G. I have not implemented any caching or anything like that, this is just a part of a Charmed/Juju OpenStack install on some older nodes. The tests were performed inside some deployed Debian 10 VMs, one with standard file storage for the disk and one with only a Ceph volume.
I previously had an environment with 3 nodes, 4x Intel DC SSDs, dual 10G SFP+ NICs (2 switches, 1 NIC in each switch port) and got a similar result using a system called PetaSAN (petasan.org). The I/O Speed metric was somewhere in the neighborhood of 70MB/s, where the RAID10 by itself performance was above 1GB/s. I ended up dumping PetaSAN in favor of Ubuntu + targetcli to present the RAID10 volumes as iSCSI targets.
Anyone have any insight into troubleshooting this setup and increasing performance without additional hardware purchases? Not that I am against that, but that has always been my go-to and I'd like to try and take a more holistic approach for when I have a setup like this colocated.
I previously had an environment with 3 nodes, 4x Intel DC SSDs, dual 10G SFP+ NICs (2 switches, 1 NIC in each switch port) and got a similar result using a system called PetaSAN (petasan.org). The I/O Speed metric was somewhere in the neighborhood of 70MB/s, where the RAID10 by itself performance was above 1GB/s. I ended up dumping PetaSAN in favor of Ubuntu + targetcli to present the RAID10 volumes as iSCSI targets.
Anyone have any insight into troubleshooting this setup and increasing performance without additional hardware purchases? Not that I am against that, but that has always been my go-to and I'd like to try and take a more holistic approach for when I have a setup like this colocated.