Proxmox + Storage

[ideabox]

Hey,

So I've built my first Proxmox VE cluster.
Ive been mainly a Mac and Windows guy and recently missed out on a job because my Linux was rubbish, So I'm diving in to train and so I first started with Unraid I liked it it was easy and intuitive but I needed 2 servers as It was cheaper to buy 2 than one high powered one. R710's from everyone here that I read.

Anyways...

Im a noob so forgive me,

I want to have a storage array connected to VE, Im thinking for getting a cheap rack chassis and building a storage array, 8x 1tb drives but how can I configure that to be joined into the clusters so other clusters can use it.

SUPERMICRO Black 2U Rackmount Server Case | eBay

The other thing is I also want it as a NAS/DAS into the VM's so I want to create a folder for example and it be available over the network to my VM's?

Thanks in advance
Phillip

 

[T_Minus]

Are you going for a HA Cluster?

They state you need this for storage:
Shared storage (SAN or NAS/NFS for Virtual Disk Image Store for HA KVM)

However, it sounds like you want to use the storage on all nodes either way so even if not doing HA you still would likely be best off to simply setup your NAS with NFS and use that for VM disk images, storage, etc....

I would look at your cluster storage and NAS either 'together' or as 2 units for HA or simple redundancy via backups.

Most around here use FreeNAS or Napp-IT which are both ZFS based file systems but are 'NAS Appliance' packages ready to go. You can run them in a VM even or on baremetal depending on your needs (with Proxmox and storage in general). With ZFS and 8x1tB you're very likely going to want a SLOG so VM storage performance is increased, and for sure lots of RAM and maybe ConnectX2 cards for 10G.

There's lots of advice and builds on NAS (FreeNas and napp-it for that matter) on this forum! I would browse builds and see what people are doing, and get more ideas, etc... depending on usage lab/testing only or production you may get away with lower-end hardware, less RAM, etc...

 

[ideabox]

Are you going for a HA Cluster?

However, it sounds like you want to use the storage on all nodes either way so even if not doing HA you still would likely be best off to simply setup your NAS with NFS and use that for VM disk images, storage, etc....

Hey,

Yes I'm planing to make it a HA cluster, Fencing is the part I don't understand so far working on it.

So maybe I have 4x 240gb SSD raid 10 for the HA & VM's and 4x 1tb raid 10 for my ''NAS''?
I don't have vast amounts of data yet so I'm thinking it would suffice my needs. At the moment my VM's have 30gb disks and that is enough for me.

The confusing part for me is how to make it available to the cluster for HA?

I got some Intel X520 cards on the way from China, Might be the fake ones but they are so cheap.

 

[vl1969]

I wonder,
Proxmox HA cluster requires 3 nodes, 2 nodes for redundancy and 3rd as witness,
can you setup a 2 node cluster and use the witness node as both a witness and NAS provider?

you know, set up a proxmox node to use as witness, and load up the freeNAS or something on it for network storage. this way it would pull a double duty.

don't have the setup myself as I only have one machine but it is an interesting idea.

 

[PigLover]

Its not exactly "2 plus a witness". Its n-way active with quorum, meaning all nodes are equals, at least (n/2)+1 of them must be available to make any changes to the cluster. In the case of 3 nodes, you need 2 running to make a quorum.

However - nobody says all three nodes need to be equals. There are lots of ways to build the third node:

• Your suggestion of the third node running on the storage node works but has maintenance complications.
• You can get whatever cheap, low power leftover PC you have around and load up Proxmox on it as the third node (e.g., a laptop that you don't use anymore because it has a destroyed screen/keyboard, etc).
• A low-end SFF system like some of the J1900 boxes that people sell on eBay as pfSense hosts ($100-$200 typical, sans disk/memory)
• There is even one creative soul out there who got a Raspberry Pi running as a quorum node by stripping proxmox of everything but Corosync and the cluster daemon (this is a true witness node design, see here: RB PI for third node).

 

[vl1969]

can you elaborate on "maintenance complications."?

 

[PigLover]

can you elaborate on "maintenance complications."?

Suppose, for example, that you need to do any sort of upgrade to "node 3", the one that supports your storage. And that upgrade requires that "node 3" be restarted. Because it also hosts your shared storage this upgrade now impacts your entire cluster and you effectively have to stop/restart all of the VMs (or at least pause them and hope that the storage state is consistent when you restart).

There are easily dozens more examples - this one is just the easiest to get across is in a few short sentences.

You are always exposed to this issue with shared storage - unless that shared storage is also built in a fault-tolerant method. But by combining the "third node" and the shared storage you've increased the number of maintenance activities that are troublesome.

Addressing this is why some people are using "hyper-converged" storage designs. A fault-tolerant storage design also requires at least 2 or 3 storage nodes, so in a hyper-converged design you build your compute hosts and storage hosts (and management host - which Proxmox already "converges") on the same set of servers. One easy example with Proxmox would be to use Ceph storage, with the Ceph control plane (monitors) and data storage (ODSs) running on the same 3 servers as Proxmox - resulting in a config where any one server can be offline with no impact to running VMs on the other two servers. Add to that a proper HA design of the VMs themselves and you can have high resiliency of your hosted services.

Of course, nothing is "free". There are complications with hyper-converged too (storage performance, complexity, extra networking to do it right, etc.). Addressing these concerns is why you might here that a Ceph/Proxmox hyper-converged system really requires at least 5 nodes to be fully functional and resilient. But that's a topic for a different day.

 

[Patrick]

@PigLover - I use 4C Xeon D nodes with 4x SSDs and 10GbE just as "extra" Proxmox Ceph nodes (three of them do not host VMs). The added benefit is that they allow a node or two to fail and the cluster keeps humming along.

 

[PigLover]

@PigLover - I use 4C Xeon D nodes with 4x SSDs and 10GbE just as "extra" Proxmox Ceph nodes (three of them do not host VMs). The added benefit is that they allow a node or two to fail and the cluster keeps humming along.

Note that I did a final edit and added this while you were posting:

Addressing these concerns is why you might here that a Ceph/Proxmox hyper-converged system really requires at least 5 nodes to be fully functional and resilient.

Given the way Ceph does resiliency you really want enough nodes that you can still get all of the nodes re-converged to a fully resilient state even with one OSD node failed. That means that with replication of 2 (i.e., 3 copies) you need 4 nodes to converge to stable. For other reasons you want the number of nodes to be an odd number...round up to 5.

But going too much deeper than that feels a bit like thread-jacking.

 

[vl1969]

giving the fact that OP builds this for learning experience
I am not sure we have to go overly deep into the possible management issues here, but I can be wrong.
also OP seams to be building things at home, so not a production 999% kind of system.
I am sure if need to update, he can tolerate shutting down the whole cluster for 5-10 min.
I would never recommend this setup for any important and/or production system.
however, that said, you are facing the same problems with shared storage too, if you need to update/upgrade shared storage and restart it, you need to shut down the cluster as well.
unless as it was point out you use a fault tolerant shared storage setup

 

[ttabbal]

Since Patrick has so many nodes he doesn't need, I am going to suggest he send them to me and I'll make sure they get used.

no? oh-well.

 

[MiniKnight]

It's almost like we need a new thread... <3 proxmox

 

[ideabox]

I wonder,
Proxmox HA cluster requires 3 nodes, 2 nodes for redundancy and 3rd as witness,

Interesting theory!

giving the fact that OP builds this for learning experience
I am not sure we have to go overly deep into the possible management issues here, but I can be wrong.
also OP seams to be building things at home, so not a production 999% kind of system.
I am sure if need to update, he can tolerate shutting down the whole cluster for 5-10 min.
I would never recommend this setup for any important and/or production system.
however, that said, you are facing the same problems with shared storage too, if you need to update/upgrade shared storage and restart it, you need to shut down the cluster as well.
unless as it was point out you use a fault tolerant shared storage setup

@PigLover Thanks heaps for your in depth coverage!!!!

So after spending my last 18 hours ready, This is the plan.

3 Nodes R710's configured with 4x 120GB SSD's each RAID 10 - CEPH , VM's, HA etc.
1 Node as a NAS for DATA also a R710 with 4-6x 2tb drives.

(I have OCD and if the rack is messy it will annoy me so I need the same looking server)

In this scenario if I wanted more storage for instance could I just add another node with drives and configure a VM with FreeNAS?

The plan is to have it for production eventually not anytime soon though.

It's almost like we need a new thread... <3 proxmox

Indeed!

 

[PigLover]

Only suggestion I'd have is to ditch the Raid and run each SSD as an independent OSD (data disk) for ceph. You're already getting 3 way redundancy with Ceph, so the mirror in the Raid doesn't really add anything but cuts your capacity in half. And you won't really get any performance benefit from the striping either. So just 12 OSDs, 4 per server, is best.

 

[Connorise]

> Are you going for a HA Cluster?

When it comes to HA cluster with 2 nodes, I recommend my clients the next options. VMware vSAN can create HA with 2 nodes + witness node( it can be cloud node). HPE vSAN can build HA with 2 nodes but performance is not the best part if we comparing with another solution.
The last solution that comes to my mind is StarWind vSAN they can build HA with only 2 nodes and due fast iSCSI and block replication they can provide good IOPS numbers.