Drive configuration for virtualization host

[ullbeking]

Hello all,

I'm configuring and testing an entry-level 1U virtualization server for deployment overseas in the next several weeks or month (this order of time hopefully). The key points are entry level and low power.

Here are the key points:

• E3-1200 v5 CPU
• 64 GB DDR4 UDIMM
• 4x LFF front-loading bays for 3.5" SATA enterprise drives

How should I arrange my drives? I have 2x large 2.5" SATA SSD's already, and I need to decide on the other two drives. I think the workable options are as follows:

1. 2x large SSD's and 2x very large spinning HDD's, arranged in two mirrored pairs
   
2. 4x large SSD's, arranged in RAID10
3. 4x very large HDD's, arranged in RAID10
   
4. 3x very large HDD's and 1x large, fast, resilient, and low latency caching SSD, arranged in RAID5

Option 1. suffers from obvious drawbacks. It means having to manually partition your application layer and data into which mirrors that you predict ahead of time. It is also less flexible to changes in drive configuration.

My favorite option is 2., based on personal preference, experience, and educated guesswork based on IOPS and latency. But I haven't measured anything and I haven't bought all the hardware yet.

Is option 3., my second personal preference, a workable one? Is it madness?

As for 4., I've pretty much already ruled it out. I am more inclined to mirroring instead of RAID5 for several reasons. I won't go into the details now unless the thread goes that way (I wish I could find the blog post that explained it better than I could right now and I will make a mental note to look for it).

I have hardware for options 1., 3., and 4. But not 2., which, as I have already stated, I am already mostly decided on.

What do you think? Thanks!!

Edit: Minor fix to RAM specification
@ullbeking

 

[Evan]

First thought,

E3 and RDIMM don’t go together, should be ECC UDIMM

 

[Evan]

2nd thought, I would say option 1 or 2

Option 1: Depends on what your storing , just using the big mirror pair for backups or bulk data then should ok.

Option 2 : my preferred option if I was deploying, for performance, low power, and as it’s overseas remote deployed then reliability would be very useful and SSD’s tend to fail less often.

 

[ullbeking]

E3 and RDIMM don’t go together, should be ECC UDIMM

Thank you for pointing this out. It was indeed merely an oversight.

 

[ullbeking]

2nd thought, I would say option 1 or 2

Option 1: Depends on what your storing , just using the big mirror pair for backups or bulk data then should ok.

Option 2 : my preferred option if I was deploying, for performance, low power, and as it’s overseas remote deployed then reliability would be very useful and SSD’s tend to fail less often.

I agree, plus these options fit my natural workflows and sensibilities.

Mainly the issue with 1. is having to manually decide all the time where something will be stored. I hadn't thought of using them for backups. In my mind, such a pair of HDD's would be either for data or backups but not both. I'd have to keep my backups elsewhere, but then they won't be local, violating the 3-2-1 principle (if this matters or not).

Option 2. leaves no room for backups but they can be stored on another host somewhere else. Similarly to my intention for the use of HDD's in Option 1.

Interesting problem and I'm going to have to think seriously about this. It adds a whole extra dimension to the design and is a constrained one. Thanks!!

 

[Rand__]

I'd say that will depend on the kind of workload you put on those?

Many vms concurrently active - you would want ssds
many vms, few active at a time - disk is fine
high io vms - ssds

what add in cards do you want to run? e3 does not have a load of free slots ... and in one U you are even more limited. Really want to spend that on a Raid controller? Or maybe rather 4 nvme drives you bifurcate on a x16 slot... ?

 

[ullbeking]

I'd say that will depend on the kind of workload you put on those?

Many vms concurrently active - you would want ssds
many vms, few active at a time - disk is fine
high io vms - ssds

Thank you @Rand__ . I have a couple of clarifications. When you say "disk is fine", do you mean "4x HDD's" or "2x SSD's and 2x HDD's"? Keep in mind I'm referring to SATA up until this point.

This is going to be small virtualization host with its own bulk storage. So there are lot of tough requirements, including how to take backups.

what add in cards do you want to run? e3 does not have a load of free slots ... and in one U you are even more limited. Really want to spend that on a Raid controller? Or maybe rather 4 nvme drives you bifurcate on a x16 slot... ?

The SSD's I was referrring in my original message are SATA. Of course I would prefer NVMe so an AOC would be needed in that case. InfiniBand connectivity would also be very useful. Looking down the line I am even considering purchasing a WIO or UIO 1U SuperServer, all as one system, which would leave me with more options for expandability.

 

[Rand__]

Examples (slightly unrealistic, just to make the point) -
Imagine you have 25 different VMs with a specific Windows version each - all identical from hostname/ip/uuid point of view due to licensing reasons. Only one can run at any time else you get errors.
=>In that case a harddrive is fine since the IO requirements at any given time are very little but needed space is large

Other example - you got 25 linux vms downloading ... ISOs ... all the time and saving to a central location outside the VM - harddrive is also fine (unless you run updates at the same time).

Less extreme example
You have 5 regular VMs which hand out data to the internet (webserver et al)
- option A - everything is cached in memory -> little disk IO after initial boot [except logs], HDD is fine
- option B - not everything is cached (due to size of data or memory limitations)-> at some time the HDDs could become overburdened with IOs...

Thats why I said - it will depend on what your usage profile is going to be...


And why 1U anyway ? 2 U would provide much more options ... or do you have to pay per U?

 

[ullbeking]

@Rand__ Thank you for your reply. I need to spend more time reading in detail, but the fact of the matter is that I am paying per U and need to keep my power usage down to a "fair use" level. It's an informal arrangement for a serious, self-funded, side experiment that I am working on. One aspect of this project is to grow out to formalizing my knowledge and practice of geo-distributed clustering. Therefore, colo costs must be kept to a minimum. I also want to keep to 1U to minimise shipping costs, and to try to practice the most economical and practical thing that I can at the moment. More to come later, and thank you!!!

 

[ullbeking]

OK, so I've just realized what the best option is, which was staring me in the face the whole time...

• 4x 3.5" large capacity SATA or SAS HDD's in the front-loading LFF bays
• 2x M.2 NVMe SSD's in a x8 dual adapter card directly attached into PCI-e

Thank you to all who have helped me figure this out.

 

[Rand__]

make sure your board does support bifurcation in the x8 slot
also on one u you'll need a matching riser card for that slot, but i am sure you considered that