esxi local storage upgrade - request advice/comments

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

poto

Active Member
May 18, 2013
239
89
28
Hello to All,

I'm looking to upgrade local storage on esxi host, and after reviewing the great posts and articles on this site I would like to

ask for any advice and comments from more experienced posters here to hopefully avoid any dead-ends as I move forward. I would

also like to give a big "Thank You" to all the contributors - this is by far the most useful site I have found for this type of

info.

The esxi host is for home use only, and current setup is as follows:

Supermicro X9SCM-iif
Xeon E3-1230v2
32g ram

Adaptec 5805 w/bbu 8x4tb Hitachi 7k4000 raid 5 (passed through to Win7 vm for media storage, backed to separate server)

Adaptec 5805 w/bbu 4x500gb Seagate raid 10 (vm backups, low i/o vms)

motherboard sata ports 4xCrucial M4 256gb (10-15 vm's)

Workload is probably light compared to most here - streaming to media extenders, vm's for family members, video

encoding/processing, usenet indexing with small db, torrents, test vm's.

One of the Crucial M4s failed, which got me thinking about redundancy on main datastores. After SSD exchange, I setup a raid 10

volume with the M4's, which performed poorly and experienced drive dropouts.

My primary goal is to have redundancy for main vm datastore using the Crucial M4's and increase read/write/iops performance as

well. I prefer a hardware raid solution vs hba + software raid.

The tentative plan is to use LSI 9271-8icc + cachevault to host 2 raid 10 arrays, 4x500gb seagate and 4x256gb M4 SSD.

Uncertainties:

1 - Am I overlooking less expensive option that delivers same performance? This is overkill for current needs, but I prefer to

invest in a stable 3-5yr solution rather than annual equipment churn. Don't currently need Cachecade, but Fastpath would benefit

2 - will the M4 hold up? Could not find long-term use examples. Budget will not permit new raid controller AND drives

3 - will the Adaptec 5805 and LSI controllers co-exist same motherboard

Thanks for any guidance you can offer...
 

mrkrad

Well-Known Member
Oct 13, 2012
1,244
52
48
1. Stable would be using read-only cachecade/maxcache/hp smartcache and sas drives (lots of them).

2. Consumer ssd's are generally buggy and i've experienced the first double drive failure in a raid-10 span after I started using SSD.

3. Sure just don't starve them, ESXi doesn't like sharing. Like 1 VM per host, if you ran 2 vm's per host - two raid controllers, you'd get perfect scaling (network too). Once esxi has to share controllers, performance goes way down in peak transfer rate.

I think any sort of VT-D is dangerous with esxi. This is why SR-IOV exists, but sadly no raid cards support this. All fiber/10gbe+ nic's present to ESXi many virtual nic's - this allows the 1 VM -> 1 ISCSI/FCOE/NIC ratio that scales. What's your network strategy?

I'm not impressed with LSI, the controllers are very buggy (firmware/drivers) in the megaraid line and their draconian DRM can put you in a serious pinch.

For instance HP uses a fixed key per server for their advanced features, on the honor system. LSI uses a physical key (could get damaged, then what?) or a license that must be activated and transferred. Hello 1990's are calling and they want your dongle protection back.

The Adaptec controllers are junk. PMC-Sierra took over around and have wonderful 71605/7805 with maxcache (same as cachecade 2) now.

Want the straight up news? I think your long term budget and goals are short sighted.

I think you should buy a DL180 G6 dual L5630 deal and stuff it with 96gb of ram, a P420/1gb FBWC. 1 or 2 SSD for acceleration, 15K SAS drives for main storage, and a couple of RE4 nearline SAS large drives (P420 can take 3 nearline 4TB and create 1 raid volume, extra speed, and realistic rebuild).

Ram is king. Ram is cheap right now. Servers are cheap. SSD's that can go the distance are not. Unsupported ssd's are problematic (except for simple read-only caching of volumes cachecade 1/hp smartcache/maxcache). Use of nearline drives is well dangerous. There is a reason you get a year warranty (usage adjusted) for consumer SSD and 1 year warranty on server nearline SAS/SATA drives. They aren't that reliable.

You may get lucky, but luck is for only the most adept or crazy (or both) folks who can deal with the RPO/RTO when everything goes tits up and you have to perhaps stand up another equivalent server and restore your world.

But that's just my opinion.

I'd say go for it, but take this one piece of advice: You get what you pay for. I figure in a year, we'll have bigger/faster/better and if I have to replace all my ssd every year, that's cheaper than having to manage 200 10K SAS drives, the power, the heat, the chassis, and the (lack of) speed.

Good luck! Run a few other options by us.

Also if you don't need fast, throw together a "slow n steady" classic server, like the DL180 G6 with P410 and regular sas drives. Put your slow vm's on that. Let them crunch. They will have that 99% uptime. Put the fast hotness needs on SSD and plan carefully. Don't throw it all on a single server man! bad idea
 

poto

Active Member
May 18, 2013
239
89
28
1. Stable would be using read-only cachecade/maxcache/hp smartcache and sas drives (lots of them).

2. Consumer ssd's are generally buggy and i've experienced the first double drive failure in a raid-10 span after I started using SSD.

3. Sure just don't starve them, ESXi doesn't like sharing. Like 1 VM per host, if you ran 2 vm's per host - two raid controllers, you'd get perfect scaling (network too). Once esxi has to share controllers, performance goes way down in peak transfer rate.

I think any sort of VT-D is dangerous with esxi. This is why SR-IOV exists, but sadly no raid cards support this. All fiber/10gbe+ nic's present to ESXi many virtual nic's - this allows the 1 VM -> 1 ISCSI/FCOE/NIC ratio that scales. What's your network strategy?

I'm not impressed with LSI, the controllers are very buggy (firmware/drivers) in the megaraid line and their draconian DRM can put you in a serious pinch.

For instance HP uses a fixed key per server for their advanced features, on the honor system. LSI uses a physical key (could get damaged, then what?) or a license that must be activated and transferred. Hello 1990's are calling and they want your dongle protection back.

The Adaptec controllers are junk. PMC-Sierra took over around and have wonderful 71605/7805 with maxcache (same as cachecade 2) now.

Want the straight up news? I think your long term budget and goals are short sighted.

I think you should buy a DL180 G6 dual L5630 deal and stuff it with 96gb of ram, a P420/1gb FBWC. 1 or 2 SSD for acceleration, 15K SAS drives for main storage, and a couple of RE4 nearline SAS large drives (P420 can take 3 nearline 4TB and create 1 raid volume, extra speed, and realistic rebuild).

Ram is king. Ram is cheap right now. Servers are cheap. SSD's that can go the distance are not. Unsupported ssd's are problematic (except for simple read-only caching of volumes cachecade 1/hp smartcache/maxcache). Use of nearline drives is well dangerous. There is a reason you get a year warranty (usage adjusted) for consumer SSD and 1 year warranty on server nearline SAS/SATA drives. They aren't that reliable.

You may get lucky, but luck is for only the most adept or crazy (or both) folks who can deal with the RPO/RTO when everything goes tits up and you have to perhaps stand up another equivalent server and restore your world.

But that's just my opinion.

I'd say go for it, but take this one piece of advice: You get what you pay for. I figure in a year, we'll have bigger/faster/better and if I have to replace all my ssd every year, that's cheaper than having to manage 200 10K SAS drives, the power, the heat, the chassis, and the (lack of) speed.

Good luck! Run a few other options by us.

Also if you don't need fast, throw together a "slow n steady" classic server, like the DL180 G6 with P410 and regular sas drives. Put your slow vm's on that. Let them crunch. They will have that 99% uptime. Put the fast hotness needs on SSD and plan carefully. Don't throw it all on a single server man! bad idea
OK, thanks for the overview of my options; you brought up some possibilities I had not considered.

It sounds like the Crucial M4's I'm using would not be reliable for write cache, I will stick with with read-only for now.

I don't have a network strategy - probably a bit out of my depth here. Most network traffic is between vm's on the same esxi host, except NAS backups and streaming to media extenders.

With respect to the Adaptec controllers, would you consider the 71605Q equivalent to LSI 9271-8icc w/cachevault and sas expander? I've not been able to find any comprehensive testing on the Adaptec, just marketing teasers.

The HP storage server sounds like a great deal, but I need to stay with single server right now.