Help me select NVMe storage for my Ceph Build

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

VMman

Active Member
Jun 26, 2013
125
45
28
Thanks in advance to all the storage experts out there.

I've been wanting to get a small Ceph cluster up for some time and finally have some budget to start spending :cool:

I want to start with a 5 node setup and use the experience gained to assist me with a deployment for the office at a later date. I'm planning to use dual socket v1/v2 intel scalable 24 bay servers (pcie-gen3) with Mellanox 100G Connect-X/4/5 NICs and 100G switches (SN2100)

I'm seeking some advice on the following:

Do I consider using newer gen NVMe SSD's like for example the Micron 7450 Max over the pervious flagship being the 9300 Max considering that my chassis can only use U2/PCIe3 4x?

I may not see any the performance benefit today but future proof my investment for newer hardware once I life cycle my current lab and can make use of PCIe4.

Alternately do I opt for the cheapest cost per TB of NVMe being something like the Intel 4510 series?

Am I limiting myself in terms of overall performance by opting to use a older gen NVMe or would this be negated by CPU/s and NIC speed should I opt to use more than 8 disks per node?

I guess this question kind of falls in-between a Ceph and a hardware section.

Are there any other NVMe SSD's that I should be considering?

Datasheet 7450/PRO/MAX Link
 

ano

Well-Known Member
Nov 7, 2022
634
259
63
ceph is still very slow, even vs older drives, but new reef etc is "getting there"

its a money thing, and what you wanna do down the line thing.

7300 = good
7450 = good
pm9a3 = usually good
p4610 = good
kioxia CD6 = good
kioxia CM6= good

on 1/2nd gen xeon, cpu/bus will limit, ceph will limit.

what is use? what do you expect?
 
  • Like
Reactions: VMman

VMman

Active Member
Jun 26, 2013
125
45
28
Thanks, use case is going to be serving up a mixed VM workload for four Proxmox hosts. At this stage I don't really have any speed expectations but I'm trying to squeeze the max performance out of the hardware that I have available.
 

ano

Well-Known Member
Nov 7, 2022
634
259
63
so basicly hyperconverged with RBDs

max performance, and ceph doesnt really mix, but it should be decent, and with newer versions of ceph, more and more performance

how many vms? any idea on workload? iops patterns?
 
  • Like
Reactions: VMman

VMman

Active Member
Jun 26, 2013
125
45
28
Sorry for the lack of detail... No the hypervisors separate servers running on single socket AMD Rome connected with 2 x 25Gbit into the same switch. I envisioned the Ceph nodes to be dedicated and not use a hyperconverged design.

For VMs I'm building this for 160 over the four hosts.
Workload is very mixed, I'd say 40-50% Web/NGINX, 30% DB and 20% Misc mostly read by my estimate.
 

Sean Ho

seanho.com
Nov 19, 2019
768
352
63
Vancouver, BC
seanho.com
5x nodes with 8x NVMe OSDs each, 3x replication, and 2x25GbE should support that workload pretty well. You might consider running test workloads and playing with multiple OSDs per drive, watching CPU and RAM load.

If the DB I/O starts to become a bottleneck, you might consider putting the DBs on a separate RBD pool with smaller object size to avoid write amplification effects.
 
  • Like
Reactions: VMman and ano

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
You definitely should run test workloads ...
As long as you dont expect peak performance for single vms, total aggregated performance should be fine
(Note this is not based on current experience but tests a few years back with just a few vms which were unsatisfactory despite nvme)
 
  • Like
Reactions: VMman