Proxmox/Ceph Distributed System - Advice Sought

WeekendWarrior

Active Member
Apr 2, 2015
341
136
43
55
I am planning a Proxmox/Ceph installation and would appreciate some advice on performance aspects of such a system. My question seems to concern Ceph but that will be installed within the context of Proxmox usage.

My goals are to have "many" servers (at least by SMB standards) running Proxmox/Ceph in my office and to have "a couple" servers running Proxmox/Ceph in a colo facility. "Many" means a dozen or more and "a couple" means one or possibly a few but not "many" because power limits in colo facilities constrain what I can do in the colo facility. The office and colo networks will be linked by a persistent VPN.

I intend to host VMs/containers on the server(s) in the colo facility, and for the VMs/containers to have full access to the copy of my files on the colo server(s). I prefer to host the VMs/containers in the colo facility because its power and network connectivity is more reliable than what I could provide in my office.

So, the advice I seek is:
1. is it practical to configure a Proxmox/Ceph system to be sure that one full copy of my files is on the colo server (for performance reasons when hosting VMs/containers there) and multiple full copies of my files are on the office servers?
2. do my goals not make sense for some basic or subtle reason?

Thanks in advance!
 

Jeggs101

Well-Known Member
Dec 29, 2010
1,520
238
63
Why not more servers in the colo?

WAN replication has always been hard for Ceph

At least you're doing this with >10 nodes. The 3 node ceph clusters are usually bad ideas.
 

WeekendWarrior

Active Member
Apr 2, 2015
341
136
43
55
Why not more servers in the colo?
I gave this approach serious consideration, because it would be far better than WAN replication, but the expense would be several thousand dollars per month which is more than I can swallow.

Admittedly, I can scale back my # of systems aspirations to reduce the colo cost, but my conclusion was that I simply could not afford to do any form of putting enough systems in the colo facility.
 

Jeggs101

Well-Known Member
Dec 29, 2010
1,520
238
63
Why several thousand dollars per month more to colo a few servers? 12 servers unless they're full out dual cpu won't run 4KW. Unless these are going in high frequency trading space a rack for $1600/ mo with more power than you'll need for 12 servers is easy including plenty of bandwidth.
 

WeekendWarrior

Active Member
Apr 2, 2015
341
136
43
55
Why several thousand dollars per month more to colo a few servers? 12 servers unless they're full out dual cpu won't run 4KW. Unless these are going in high frequency trading space a rack for $1600/ mo with more power than you'll need for 12 servers is easy including plenty of bandwidth.
I like the sound of what you're suggesting. If you can point me to a specific example in Nor Cal I'd appreciate thst.