Starting small with Ceph storage

[Veidit]

Hi

I am looking at a small low latency Ceph storage that I can expand later and I need to check if what I am thinking of is a good idea or I need more hardware to start with.
In my case I need fast storage with low latency and I am looking at using RoCEv2 NFS between clients and storage.

I was hoping to be able to start with one Ceph server and one nfs client and then first increase with more disks and later with more nodes, in a way of a POC but then use the same hardware to extend the storage with.
For storage I am looking at NVME storage and a ConnectX-5 or 6. Perhaps this machine: 1114S-WN10RT | 1U | A+ Servers | Products | Super Micro Computer, Inc.
Running with a SATA DOM for OS and starting with either 4 or 6 disks for Ceph.
For the client I am looking at in a way the same server and NIC but then either SATA DOM or running on a local (or two) NVME.

Yes, there is a database coming to this later but first I want to solve this and yes the database runs on local disk.

So my question, can I use Ceph like this and then expand with more nodes and disks as I need or do I need to "buy it all" at once?

 

[iGene]

In my case I need fast storage with low latency and I am looking at using RoCEv2 NFS between clients and storage.

As far as I know, the newest version of Ceph drop RDMA support, while in older version RDMA support is not in stable release, you will have to build the binary from source. I would say if you need low latency is your priority, Ceph isn't your first choice.

Still, if you want to use Ceph, I suggest starting with at least three nodes and replicate data across those 3 nodes. This will eliminate a lot of pain when you try to scale out at first.

If you start with one node you will have to edit the crush map to replicate between OSDs and not nodes. And once you start adding OSD nodes to 3 and want to replica between node, a rebalance will be needed to redistribute the data between nodes. You will probably have to adjust some settings to reduce the impact of performance during rebalancing.

So the short answer is yes, you can start with one node, but I would not recommend doing so.

 

[Veidit]

As far as I know, the newest version of Ceph drop RDMA support, while in older version RDMA support is not in stable release, you will have to build the binary from source. I would say if you need low latency is your priority, Ceph isn't your first choice.

Still, if you want to use Ceph, I suggest starting with at least three nodes and replicate data across those 3 nodes. This will eliminate a lot of pain when you try to scale out at first.

If you start with one node you will have to edit the crush map to replicate between OSDs and not nodes. And once you start adding OSD nodes to 3 and want to replica between node, a rebalance will be needed to redistribute the data between nodes. You will probably have to adjust some settings to reduce the impact of performance during rebalancing.

So the short answer is yes, you can start with one node, but I would not recommend doing so.

Thanks! What would you recommend instead? NFS(v4) and RDMA/ROCE are my requirements. I guess I can build something with ZFS or even with a hardware RAID card.