Proxmox/Ceph Distributed System - Advice Sought

[WeekendWarrior]

I am planning a Proxmox/Ceph installation and would appreciate some advice on performance aspects of such a system. My question seems to concern Ceph but that will be installed within the context of Proxmox usage.

My goals are to have "many" servers (at least by SMB standards) running Proxmox/Ceph in my office and to have "a couple" servers running Proxmox/Ceph in a colo facility. "Many" means a dozen or more and "a couple" means one or possibly a few but not "many" because power limits in colo facilities constrain what I can do in the colo facility. The office and colo networks will be linked by a persistent VPN.

I intend to host VMs/containers on the server(s) in the colo facility, and for the VMs/containers to have full access to the copy of my files on the colo server(s). I prefer to host the VMs/containers in the colo facility because its power and network connectivity is more reliable than what I could provide in my office.

So, the advice I seek is:
1. is it practical to configure a Proxmox/Ceph system to be sure that one full copy of my files is on the colo server (for performance reasons when hosting VMs/containers there) and multiple full copies of my files are on the office servers?
2. do my goals not make sense for some basic or subtle reason?

Thanks in advance!

 

[Jeggs101]

Why not more servers in the colo?

WAN replication has always been hard for Ceph

At least you're doing this with >10 nodes. The 3 node ceph clusters are usually bad ideas.

 

[WeekendWarrior]

Why not more servers in the colo?

I gave this approach serious consideration, because it would be far better than WAN replication, but the expense would be several thousand dollars per month which is more than I can swallow.

Admittedly, I can scale back my # of systems aspirations to reduce the colo cost, but my conclusion was that I simply could not afford to do any form of putting enough systems in the colo facility.

 

[Jeggs101]

Why several thousand dollars per month more to colo a few servers? 12 servers unless they're full out dual cpu won't run 4KW. Unless these are going in high frequency trading space a rack for $1600/ mo with more power than you'll need for 12 servers is easy including plenty of bandwidth.

 

[WeekendWarrior]

Why several thousand dollars per month more to colo a few servers? 12 servers unless they're full out dual cpu won't run 4KW. Unless these are going in high frequency trading space a rack for $1600/ mo with more power than you'll need for 12 servers is easy including plenty of bandwidth.

I like the sound of what you're suggesting. If you can point me to a specific example in Nor Cal I'd appreciate thst.