This partfor vsan replacement, we plan to run proxmoxcluster and let proxmox handle storage.
Does this mean there's no option to avoid rebalancing while you maintain one node? As an example, with MooseFS you can put a node into 'Maintenance Mode' and the cluster does not care at all and no rebalancing is happening. The Cluster just assumes the node is ok. Otherwise, I would gladly combine storage and compute in nodes for efficiency purposes. The extra NIC is not that big of a thing I guess.... (You can run it the other way where ceph is added to every node.. but you need the extra nic for every node.. and taking a node down for work... means ceph rebalancing generally as you'd loose OSDs)
Yes you can do the same thing with ceph.Does this mean there's no option to avoid rebalancing while you maintain one node? As an example, with MooseFS you can put a node into 'Maintenance Mode' and the cluster does not care at all and no rebalancing is happening. The Cluster just assumes the node is ok.
What's your reason to go dual socket but low core count? I always came to the conclusion that single socket is more cost efficient unless you want to have more than 64 cores or need more than 512GB of RAM (1TB can also be possible for acceptable cost with 2DPC)dual 7313 epyc milan 16c/32t each
512gb ram
Totally understandable but for EPYC I always came to different results.For me at the time it was cost.. picking up used DL 380's got me dual socket. more ram slots (cheaper ram at smaller sizes) hot swap power supplies and enterprise class ILO all on the cheap..then dropped in some 10c/20htread cpus in each socket. (cheaply as its older hardware). I was in to enterprise grade nodes for around 400$ each at the time. (not counting the SSD).
4800 is kinda pointless, except for that, it will be very nice.here is the set up i'm thinking about building.
40G mellanox switch two switches with mclag
6 servers in total
dual 7313 epyc milan 16c/32t each
512gb ram
1 p4800x
4 p4610
40g mlx3
looking for at least two failures to tolerate, 3 would be a plus.
based off of the 6.4tb drives, 3 replicas means i could get a comfortable 30-40TB with 3 failures.
initial data migration is about 20TB maybe reaching 30TB in two years.
the systems have a total of 192 cores, total needed cores for migration from standalone boxes are about 130 cores, there is more than enough memory to cover those.
there's a possibility i can bring up a 7th node if that'll help.
trying to run around 10-15 vm's right now, mostly windows with stable and protected storage.
reccomendations for project theese days are changeing, usually people just vlan stuff, instead of separate (on 2x100 and high bw)On the ceph backend network, if you don't have the backend network its very easy to overwhelm your entire network if you have any type of unplanned rebalance event (some ssd dies for instance ) I don't build ceph without one.. The people that wrote it knew what they were doing.