Seeking recommendations for the most reliable server

[lpallard]

As a follow up of my threads in the Controller section, I am posting this thread because soon I will be rebuilding from scratch my server and I want to do things differently. Not because what I was doing didn't work, but because it wasn't bullet proof and required tons of unnecessary work to maintain and use..

So far what is known:

OS will be CentOS 6.5 or whatever current version will be available under stable release
Hard drives will be two HGST Ultrastar 15k600 15000RPM SAS2 300GB drives in hardware RAID1
RAID controller will be IBM ServeRAID M5016 with 1GB Cache and BBU
Motherboard will be Supermicro H8DCL-iF
CPU's will be two Opteron 4334 6cores each at 3.1GHz
RAM will be 64GB DDR3-1800 Kingston ECC

Other than that, I am interested in having a VM for this server instead of a base metal install like I was using before.

Other topics of interests are:

*Partition scheme (/boot alone? /tmp alone? /usr alone? /var alone?) and what are the pros/cons of segragating them VS a unified / partition?

*Which FS to use on each partition.. Other than standard system partitions (/boot, /, /usr, /var etc..) I will have a large RAID5 partition for movie storage. What would be the recommended FS for each one of these partitions? I am asking because I know that each partition sees a different file usage and workload, for example:

/boot is only used at boot time and requires the simplest configuration possible to be reliable and bullet proof
/var sees a lot of activity (system & application logs, PID files, etc) so very dynamicby nature
/tmp sees lots of temporary files and must be fast
/usr is mostly static since it will host the binaries (so more reads than writes IMO)
/whatever (the large RAID5 partition) will see seldom writings when I put a new movie, but other than that, will only do reading but requires to be reliable against data corruption..

You see where Im heading to!

*On which partition should I put the MySQL databases? I will use InnoDB and the server sees moderate to heavy SQL loads... Cant comment more because I am not an expert at it. My SQL server hosts DB's for Kmymoney and Amarok, web apps such as SAB, CouchPotato, SickBeard, and other web pages and web services, and probably the biggest demand on the SAL server is the NN+ indexer I run 24/7.

*Is there any interest in LVM? I understand the concept but woudl it be interesting for me to use? Move & resize partitions, etc...

*I am leaning toward Proxmox even if it is NOT a type 1 HV.. Any problems in implementing the above goals under proxmox?

The general idea is to have the most bullet proof server. If something crashes or dies, I can restore and get back in action quick.

I am VERY interested in hearing what you guys have to say!! Im sure this will be a good discussion topic for the most reliable server..

PS: Dont recommend different hardware because I already have all the hardware. Im just looking to configure the machine differently

Thanks to ALL who can weight in!

 

[Mike]

Proxmox is fine, no matter what type of hypervisor it is. VZ containers could give you an edge for some IO intensive stuff.
However, Proxmox is Debian based, not CentOS. You may want to settle on one distribution, or atleast family.

Any filesystem will do, although BTRFS doesn't work very well with its default configuration in VM-workloads. If you want to easily restore images then it would be a good idea to do VMs in disk-images, preferably QCOW, and live with the slight overhead. Or roll your own KVM and containers on a CentOS/Debian installation.

 

[lpallard]

Thanks for replying!

I was aware that Proxmox was only a wrapper distro (debian based) of different virtualization/containment technologies, namely KVM & OpenVZ..

If it was a single OS system (the way all of my machines were configured & built so far), I would have a lot less questions because I would simply partition the drives, format them with a good / relevant FS, and install... Now adding the Proxmox variable into the equation, I am a bit confused..

When I install proxmox, does it matter which FS I use and which partition scheme I opt for? My understanding is that a VM under proxmox is more or less only a huge file stored on the storage media. If true, then whats the point of spending even 2 seconds discussing about filesystems and partition schemes as anyways everything will reside in a single file?

Maybe I am better off forgeting about virtualization alltogether.. My intention with Proxmox (or VMware or anything like that) was to run different machines simultaneously (a ubuntu server, my pfsense firewall, another linux based VM and a windows XP VM). I understand that CentOS (the OS I have decided to invest in) does virtualization. COuld I simply install CentOS baremetal (right on the hardware), use its virtualization technologies to create and use the required VM's and that would be it?

Would that work?

That way, I would be able to properly use all of the hardware's capabilities and performance...

 

[cptbjorn]

Yes, you can do it all w/ CentOS installed on bare metal, either with GUI or CLI. A really good book on this is the RHCSA study guide.

 

[OBasel]

The big thing you'd get with Proxmox is a web interface. convenient if you have a cluster. Another option is to just use a bare metal HV like ESX, Xen or Hyper-V. You get a graphical interface and easy setup. CentOS can do this but I always find it a bit harder

 

[Mike]

The more I read about hardware support and required licences for management, the more i see KVM and containers as a better alternative being arguably more energy efficient, more up to date if you want and more flexible. There are the complete packages like Proxmox and Ovirt (there you have your CentOS), or graphical interfaces like ganeti by the Google guys and hell, even cloudmin for a single node.

 

[lpallard]

So Ovirt is more or less a management platform that I would install within a baremetal OS (lets say CentOS in my case) to remotely manage VM's? Ovirt wouldn't be doing the actual virtualization but CentOS's KVM & OpenVZ would be?

That would be like: Hardware > CentOS + Ovirt > VM's ??

If its accurate then I understand that I can virtualize either using a type 1 or 2 hypervisor. ESXi is a type 1 HV so all the VM's resides at the same level, right after the HV. Type 2 HV's like Proxmox or Ovirt , the VM resides within the HV (being the OS, proxmox debian based or Ovirt, an extension to the OS)..

In any ways, all the questions I was asking about partition scheme and filesystems are irrelevant with virtualization as the virtualized machine will reside in a single image file... What matters to me is then how to partition and format the drives at the hypervisor level, let it be proxmox or CentOS+Ovirt...

Is that image accurate??
Image - TinyPic - Free Image Hosting, Photo Sharing & Video Hosting

 

[Mike]

More or less yeah. Containers are as close to native as you can get a "VM" on these systems. I wouldn't worry about Type 1 and 2.
If you are not worried about disk encryption then any partition layout is fine. I would advise the use of LVM for resizing and possibly using logical volumes for VMs if you want the most out of it. The FS on top of that is totally up to you, but like i said; BTRFS is maybe not what you want for VMs atm.

 

[lpallard]

More or less yeah. Containers are as close to native as you can get a "VM" on these systems. I wouldn't worry about Type 1 and 2.
If you are not worried about disk encryption then any partition layout is fine. I would advise the use of LVM for resizing and possibly using logical volumes for VMs if you want the most out of it. The FS on top of that is totally up to you, but like i said; BTRFS is maybe not what you want for VMs atm.

Mike, I suppose even using Ovirt on Centos cannot block me from using containers as well if need be..

I will not worry aboyut type 1 or 2.. Same for encryption. In my case, its just one more layer of complication that will go wrong (remember Murphy's law?!).

As for partitioning the RAID1 array, I will probably use Centos's recommended partitioning scheme. LVM may be a good idea to use for resizing partitions. Can you (or someone else on the forum) present a real life scenario involving virtualization and LVM's?

Before all that (in the good ole days of a slackware machine with 6 drives) I was using LVM to store my movie & music collection. As the logical volume was getting tighter in space, I was putting a new HDD, expanding the LV and that was it! I got rid of that when I realized that I was losing a drive per quarter (3 or 4 months) and that if I had lost a participating drive of the LV, I would have lost the entire volume.. Then I moved to RAID5.

I installed Centos & ovirt in a VM under virtualbox on my desktop machine. All went well (other than the Centos way of doing things which I am not used to) but I couldnt get past the host creation. I wonder if it has to do with the fact that I am not using a server CPU to begin with, and maybe VBox doesnt pass the proper virtualization extensiosn to the CentOS VM... Doesnt matter, my SAS drives and PSU should be here within a week or 2 so I will have real server hardware to try for real..

 

[Mike]

LVs allow you to make snapshots, although some disk-images let you do that too.

 

[masterdam79]

At work I use ESX with LVM especially to expand the volumes online without rebooting. That's a real life scenario for ya ;-) I have never seen my LVM volumes schrink like you described.
At home I really like to use proxmox. I run it as a 3 node cluster and all CT's have different functions. One runs a shared hosting server while some other CT's function as media server, NFS or Samba server to stream media to my media center setup.

I tend to setup my proxmox nodes as default as possible to make clustering easier whenever I'm replacing a node.

With regards to the partitioning layout I can only say that it's irrelevant when using Proxmox CT's. When using proxmox VM's you can isolate the VM's a bit more and use different file systems.

At work we for example umount /boot after boot (don't forget to mount it again when doing yum update) and the /tmp has the parameter noexec in fstab.

 

[masterdam79]

A combination of hardware (hot-swappeable) RAID and LVM is IMO the best combination for stability and flexibility.

 

[lpallard]

A combination of hardware (hot-swappeable) RAID and LVM is IMO the best combination for stability and flexibility.

masterdam79, really interesting feedback you bring here!

Just for my own education, with vitrualization platforms such as oVirt, Proxmox, ESXi, etc.. when you install an OS in a VM is it installed just like if it was bare-metal? In other words, lets say I create a 100GB virtual disk in Proxmox or oVirt, and then I install Slackware, Fedora, CentOS, etc... will I be able to partition the same way as if I was installing on a physical 100GB hard drive?

Also, I have been increasingly intersted by virtualization and hypervisors.. Searching for stability, reliability being paramount to me, wanting a financially free solution (I dont really care for free as in free software), and not wanting a solution that will put roadblocks (RAM ceiling, features being blocked in free editions, etc):

-ESXi is out because requires a windows client to use vSphere. Also I understand it is somehow restricted for free usage..
-Containers are nice but dont allow to run Windows VM's or FreeBSD (so no pfsense router)
-HyperV is microsoft... 'nuff said!

So whats left is Proxmox and oVirt (using KVM and Containers). In the end, it will be on of them.

masterdam79, you prefer proxmox over ovirt? Can you quickly compare them? Is proxmox more stable & reliable than CentOS+oVirt?

With oVirt, if I understand well, you have an extra layer before the hypervisor:

CentOS [BASE OS] > oVirt [HYPERVISOR] > Virtual machines

Proxmox [HYPERVISOR] > Virtual Machines

Or maybe Proxmox is identical to oVirt+CentOS, Debian the "CentOS" of my "BASE OS+HYPERVISOR" scheme..

 

[masterdam79]

I'll answer a bit more in detail tomorrow GMT+1 but I have no experience with oVirt.
Proxmox delivers two virtualization schemes.
OpenVZ (containers) and KVM/qemu (VM). I use both and decide which scheme suits the task at hand most. Also it's completely free except for support of course. No limitations whatsoever.

 

[masterdam79]

With OpenVZ containers I like to use bind mounts to mount disk on the hardware node inside the CT.
Read/write is as fast as can be because there is nothing in between.
For example I have a disk with movies but my samba server is virtualized so that I only have to backup my fstab and samba CT.

Whenever I run into trouble and have to reinstall the hardware node out of the cluster, I can be up and running in no time.
Downtime is 0 for my shared hosting server as it fails over to another hardware node.

 

[masterdam79]

I'm on a roll..
Answering your first question in your last post.
When using Proxmox VM's (KVM/qemu) the guest OS is behaving like a bare metal.
When using ESX as well.
When using OpenVZ it's a little more intertwined but you'll be able to run different distributions, just no windows obviously.

 

[lpallard]

OK just so we're clear, does Ovirt requires 2 physical hosts to work as intended? I mean, I couldn't setup a single physical server to be a virtualization host. Reading documentation and SEVERAL websites, it seems that ovirt requires a machine to be the management machine, and another(s) to be hypervisor ones...

Im also getting SSH problems when installing hosts (the truth is that I couldn't get past the host installation)... Also, lots of packages had to be installed to sustain a chance to work.

A wiki page on ovirt's webpage seemed to describe how to setup ovirt to be a REAL virtualization machine:

Feature/AllInOne

If I understand, ovirt is NOT meant to run on a single machine (ovirt-engine & VSDM on the same machine) unless special packages are installed.

Anyways, I found it confusing and too complicated for nothing.

Until I can see otherwise, Proxmox is the only one left.

 

[RTM]

OK just so we're clear, does Ovirt requires 2 physical hosts to work as intended? I mean, I couldn't setup a single physical server to be a virtualization host. Reading documentation and SEVERAL websites, it seems that ovirt requires a machine to be the management machine, and another(s) to be hypervisor ones...

Im also getting SSH problems when installing hosts (the truth is that I couldn't get past the host installation)... Also, lots of packages had to be installed to sustain a chance to work.

A wiki page on ovirt's webpage seemed to describe how to setup ovirt to be a REAL virtualization machine:

Feature/AllInOne

If I understand, ovirt is NOT meant to run on a single machine (ovirt-engine & VSDM on the same machine) unless special packages are installed.

Anyways, I found it confusing and too complicated for nothing.

Until I can see otherwise, Proxmox is the only one left.

I installed Ovirt on a test machine a month or two ago in the all in one "mode", it works very well.
My understanding is that all in one functionality is rather new, but unlike Openstack it does not appear to be a "for devs" function. (EDIT: I am sure with the proper measures you could use Openstack AIO in production, but most guides I have bothered reading, suggests against it. But don't hold it against me if things have changed.)

The difference between all in one and regular is one step, which you can find in the installation instructions:
Download

Ovirt is not complicated, but you should definitely try it before you put it into "production" (and don't bother with VMs it will detect that you are in a VM and abort).

 

[BThunderW]

Since we're talking virtualization, does any one know of a ESX alternative that will boot a node from USB/Flash? ProxMox will install to USB but since it writes / swaps to local storage it'll kill the stick in no time.

 

[dataoscar]

I tried using oVirt for AIO and it did not go too well. I wanted to passthrough an HBA but found the process cumbersome and unsupported.

It got me thinking that if things broke debugging would have been a nightmare.