Best RAID controller for new virtualization setup

[lpallard]

Hi, you guys have helped me a lot when I picked up my M1015 and flashed it to IT for HBA use..

Now that I am about to move my local server to a virtual setup, I wonder about RAID adapter for the VM's storage. ESXi setup instructions are recommending installing the VM's on a RAID 1 array for redundancy, etc..

As of now, I am running two RAID1 arrays (Linux mdadm), one for /boot, one for / off of the mainboard controller, and another RAID5 array with lots of data (approx. 6TB) using 6 HDD's from the M1015.

I am wondering if I should just pass-through the two HDD's used in the RAID1 arrays to the VM's and reconstruct the RAID arrays in the VM's, or should I simply get a RAID controller, setup the RAID1 array at the controller level to pass-through only the resulting drive and be transparent to the VM's?

For the price of a good second hand controller (took a quick ebay drive and looked around), is it worthwhile to buy such controller and use it for only 2 drives? Most of the dual SAS port controllers can work with 8 SATA drives. Assuming 2 drives for the VM's, there would be 6 drives left (the RAID5 array).

-Which controller would you recommend to me? M5014, M5015 or M5016? Another model/type?
-With that controller, can I do RAID5 or 6? Do I need to spend another extra 500$ for a advanced key?
-Can I create 3 RAID arrays using drives from both SAS ports?
-What about those controllers (m5015, or M5016) from Taiwan or Hong Kong? Often I see them about 20% cheaper than the US counterparts.. Are they fakes? DO they work as intended?
-Is the BBU absolutely required or just a nice thing to have? I am using a UPS for this server if that matters.
-Are SAS drives really worth it? For 2X 2TB SAS drives, I would have to shell out $400CAD which is a LOT for me right now.. I would go for 1TB drives but Im already using 1.2TB out of 2TB of the RAID1 arrays and expect to outgrow this some time soon... I may have to revisit my storage strategy..

In regards to the M1015 I already have in this server, I'd like to keep using it either as HBA or for RAID. Is it any good in RAID?

I am using a Supermicro H8DCL-IF motherboard.

Thanks a lot!!

 

[lpallard]

OK some research later, this is what I've found. I am a bit uncertain but I guess keeping up the search will lead to an answer?!

I saw on this forum user mrkrad mentioning that the M5014 with its 256MB cache was not the best due to cache choke. Anyone can explain in layman's terms??

http://forums.servethehome.com/great-deals/1075-ibm-m5014-w-lsiibbu08-$149.html

They are buggy. LSI controllers with 256meg are 32bit +8ecc , the 512 are 64bit + 8 ecc , so the cache is choked big time. Same as the P400/P410 hp smartarray. avoid 256meg cache always.

Also fastpath doesn't work on the M5014. Only cachecade. So buy that performance key and no fastpath.

LSI will not help you.

So that would pretty much eliminate the 5014 and equivalents of the SAS2108 with 256MB Cache.

Then, I was leaning toward the M5015 with its 512MB Cache, until I found:

http://forums.servethehome.com/raid-controllers-host-bus-adapters/397-ibm-m5015-experiences.html

RAID 0, 1 and 10 or JBOD get a M1015 this does it soooo much better
The extra layer that the SAS2108 brings between HDD and OS slows real world performance

RAID 5 and beyond the M5015 will excel as the SAS2108 eats XOR for B'fast

So can anyone confirm that the M1015 is better than the M5015 for RAID1?? I reckon that the M1015 doesnt have a BBU or cache, but are these essentials? Like I said, my server runs behind a UPS.

 

[Chuckleb]

I can't answer the questions about cache bottlenecks, but the card based BBU is important if you want to use the cache on the card. The cards can use the cache in write-through or write-back mode. Write-through writes data directly to disk when it is told, write back caches the write and tells the OS that out is written and safe. Say you crashed, blue screen or power outage, the data in cache not written could be lost. The BBU would supply power while things stabilized (system reboot). A whole system UPS could not tell the card to keep the data in cache.

So no BBU, I recommend to do write-through. LSI recommends that too.

 

[mobilenvidia]

If data integrity is essential get a UPS and SAS drives, you risk loosing data with anything less.

RAID1 is simple RAID, just write the data to 2x drives (mirror) so nothing of any prowess needed.
Caching will speed up writes (up to the cache size) and small often read data.
But overall it won't speed up, you are limited to the speed of the drives.

RAID5 means crunching numbers (XORing), a caching controller will always be faster at this.

 

[lpallard]

If data integrity is essential get a UPS and SAS drives, you risk loosing data with anything less.

RAID1 is simple RAID, just write the data to 2x drives (mirror) so nothing of any prowess needed.
Caching will speed up writes (up to the cache size) and small often read data.
But overall it won't speed up, you are limited to the speed of the drives.

RAID5 means crunching numbers (XORing), a caching controller will always be faster at this.

Thanks guys for replying!

With what you said in mind, I'd rather start by describing the current scenario:

-2X 2TB attached to the mobo's SATA ports, assembled as a 2TB RAID1 array, with 66% full
-3X misc sizes HDD attached to the modo's SATA ports assembled as RAID0 for temp scratch disc (will be replaced by SSD)
-6X 2TB attached to the M1015 (IT mode) assembled as a 10TB RAID5 array with 48% full.

Out of the 2TB RAID1 array, around 25GB is used for the server's OS and packages. This is highly important and requires the highest (for the $$) level of protection against data corrption and loss.

As for the remaining 1.2TB of the 2TB array, this is used for SQL databases, hosts backups, IT maintenance tools and packages, mirrors, document management systems, Drupal, etc. This is the server's data more or less. IMO this requires the same level of protection against data corrption and loss hence why I put it on the same RAID 1 array as the OS (/)

Regarding the 10TB RAID5 array, this is mostly files that can be retrieved from sources, machines backups, some backups of my DVD collection, software mirrors, etc.. The RAID5 array provides a certain degree of protection against loss if a drive dies to eliminate the need of recreating the entire content which took several years to fill up at first.

So if my data integrity was paramount, I understand that it may be a good option to get a RAID adapter with a BBU and cache, and 2X SAS drives for a RAID1 array(s)... With offline backups performed to a hotswap cluster, would you think this is a very decent strategy ?

With this in mind, The current server has to be reworked to:

-store VM's (which will of course include the current server's OS)
-provide safe and sound storage for the 1.2TB of crucial data
-either reuse an existing linux software RAID5 array of 6 2TB drives or transfer the data (approx. 6.25TB) to a different storage scheme. It was often pointed out that RAID6 arrays with 6+ drives is really dangerous due to the possibility of a second drive dying while the array is rebuilt.. Hence the choice for a RAID5 array until I outgrow the 10TB cap. of the existing array.

I would lean toward the following configuration, provided it makes a certain sense..

M5015+BBU + 2X 2TB = VM's + most important stuff (Mysql, etc)
Passthrough all motherboard SATA ports to the main VM (the current server's OS) and assemble the RAID0 array until SSD is purchased
Passthrough the M1015 (while in IT mode) to the main VM and assemble the RAID5 array. To access this array I will have to access it with NFS.

Make sense? Any improvement suggestions?

 

[Rhinox]

I would lean toward the following configuration, provided it makes a certain sense..

M5015+BBU + 2X 2TB = VM's + most important stuff (Mysql, etc)
Passthrough all motherboard SATA ports to the main VM (the current server's OS) and assemble the RAID0 array until SSD is purchased
Passthrough the M1015 (while in IT mode) to the main VM and assemble the RAID5 array. To access this array I will have to access it with NFS.

Make sense? Any improvement suggestions?

Seems quite good to me. Just and idea: If you do not have M5015 already or you can not get it for good price, consider also M5016 which has these extra-benefits:

- supports raid6 (for M5015 you'd have to get extra feature-key)
- has capacitor instead of battery (should hold on longer than li-ion battery)
- has dual-core cpu (M5015 only single-core)
- has bigger & faster cache 1GB/DDR3-1333 (M5015 has only 512MB/DDR2-800).

IMHO especially cache might be of importance in your use-case. Remember, ESXi does NOT do any disk caching/buffering, unlike any other modern OS. All disk-caching is done only by controller, so the more cache on controller, the better!

BTW, I was looking for M5015 recently and to my surprise I got brand new M5016 for less money than I would get M5015 (eBay)...

 

[lpallard]

Seems quite good to me. Just and idea: If you do not have M5015 already or you can not get it for good price, consider also M5016 which has these extra-benefits:

- supports raid6 (for M5015 you'd have to get extra feature-key)
- has capacitor instead of battery (should hold on longer than li-ion battery)
- has dual-core cpu (M5015 only single-core)
- has bigger & faster cache 1GB/DDR3-1333 (M5015 has only 512MB/DDR2-800).

IMHO especially cache might be of importance in your use-case. Remember, ESXi does NOT do any disk caching/buffering, unlike any other modern OS. All disk-caching is done only by controller, so the more cache on controller, the better!

BTW, I was looking for M5015 recently and to my surprise I got brand new M5016 for less money than I would get M5015 (eBay)...

Interesting!

The M5016 I saw on ebay right now are all 600$+.. Above my budget. I will continue looking for the 5016 but in the meantime, the 5015's I saw that were the "best deals" all comes from Asia. Like I said in my first post, anything I should be aware if ordering from a seller located in Taiwan, Hong Kong, etc ??

For now, the interesting Fleabay listings are:

NEW 90Y4304 IBM Serveraid M5016 SAS Sata Controller FOR IBM System X | eBay
-EXPENSIVE: approx $533CAD+shipping+import duties
-Apparently new
-Seems tu use an external battery pack (Tecate) ==> Is this the capacitor bank you refered to?
-Fleabay ad doesnt say how much cache, etc...

IBM Serveraid M5015 SAS Raid Controller With LSI BAT1S1P Battery | eBay
-Has a used battery (to avoid IMO, I'd rather get the controller then a brand new batt)
-Price in budget

NEW IBM M5110 Serveraid SAS Sata Controller FOR IBM System X | eBay
-From Taiwan ??!
-No optional cache installed, if finding a 1GB cache card to add on for less than a 5016, then this may be worthwhile
-M5110 instead of M5015 or 16..
-Seems to use PCIE 3.0 (my motherboard has only 2.0)
-No advanced feature key required

NEW IBM M5110 W 1GB Cache Serveraid Controller 81Y4481 90Y4449 Better M5015 | eBay
-price in budget but suspiciously low (or all the others are rip offs)
-Same as previous but with cache card (1GB)

IBM 46M0829 Serveraid M5015 SAS Sata Storage Controller Bnib | eBay
-Price OK
-Apparently brand new
-With battery
-Ad title says M5015 but description saya M5014... I will ask the seller for clarifications if need be..

So what do you guys think!? really to a noob like me, the best option right now on fleabay seems to be the M5110 with the 1GB cache card but what about battery or supercapacitors??

 

[Rhinox]

Hm, I bought a brand new M5016 for 250€ (german ebay), that's ~370CAD. It has listing price 269€, but seller accepted my 1st offer (could maybe try even less, he seems still to have plenty of them). I'm not sure what shipping/import charges might be for you. BTW 2 weeks ago one was sold for 211€, damn I missed that one...

Used battery is as good as no battery. A new one holds capacity for about year, then starts dropping quickly. And new BBU is quite expensive. Might be as much as half of the controller-price itself.

5110 is good controller if you add cache and battery/flash backup unit to it (but raid5/6 only with another upgrade-key, which makes it actually 5015/5016). If all you need is raid0/1/10, you can go for it.

That last one seems to me to be really M5015 if I can trust the picture, because M5014 comes without BBU (and has only 256MB cache). Also IBM part-number is correct, 46M0829 is for M5015 (46M0916 for M5014).

 

[lpallard]

OK I think I will next to that:

Contact the German Seller to ask how much it would be to ship to Canada and ask Border Agency for the duties.. Sometimes, we get really ripped off in Canada for import duties..

Then I checked the ebay add for the 5110 once again, and found one more picture in the ad showing that the 1GB cache card comes with the controller but the battery pack doesnt. I am confused because I thought that the 5016+ no longer used the battery BBU but had "supercapacitors"?.. From IBM Redbooks, I understand that the 5016 no longer relies on BBU (has supercap instead) but reading the M5110 page they dont talk about supercapacitors leaving me to believe that only the 5016 uses supercaps instead of batteries.

Can anyone confirm or clarify the BBU/battery/supercap for the 5015-5016-5110?

So basically, two excellent solutions:

5016 + 1GB cache (apparently comes with supercap, to be confirmed) + No separate BBU

5110 + 1GB cache + separate BBU

 

[Rhinox]

I'm pretty sure M5015 comes with BBU and M5016 with capacitor. It is a standart part of delivery. I have seen both packages as they came, never opened, still with IBM-seal on it. So unless seller opened the package and took battery/capacitor away, it must be in the box. The above mentioned german seller delivers it as complete unopened box. Inside is controller itself, capacitor (with cable to attach it), small adhesive sticker to fix capacitor somewhere in the case, users' guide and CD. Neither sas/sata-cables, nor low-profile bracket was there.

In case of M5110, battery/cache is optional and must be ordered separately as upgrade-kit. And you need both cache and battery. If you count it together, then M5110 plus cache plus BBU (still without raid5/6 support) might cost more than M5015/M5016...

So in standard packages M5016 comes with super-capacitor, M5015 comes with battery (does not support supercap) and M5110 comes without battery/supercap (supports only battery).

 

[lpallard]

I'm pretty sure M5015 comes with BBU and M5016 with capacitor. It is a standart part of delivery. I have seen both packages as they came, never opened, still with IBM-seal on it. So unless seller opened the package and took battery/capacitor away, it must be in the box. The above mentioned german seller delivers it as complete unopened box. Inside is controller itself, capacitor (with cable to attach it), small adhesive sticker to fix capacitor somewhere in the case, users' guide and CD. Neither sas/sata-cables, nor low-profile bracket was there.

In case of M5110, battery/cache is optional and must be ordered separately as upgrade-kit. And you need both cache and battery. If you count it together, then M5110 plus cache plus BBU (still without raid5/6 support) might cost more than M5015/M5016...

So in standard packages M5016 comes with super-capacitor, M5015 comes with battery (does not support supercap) and M5110 comes without battery/supercap (supports only battery).

Happy New Year to all!!

So I have contacted the German seller and he indicated that the shipping fees to Canada would be around 50 Euros (73CAD) so still a bit cheaper that the other ebay M5016 I see from the US..

There are no import duties for computer parts in Canada. That helps !

Rhinox, if ordering from this German seller gives you a brand new M5016 controller with the supercapacitor but what about the cache card? Can you confirm it was included with the controller?

Also, are there any compatibility issues with certain controller models and certain motherboards? How can I be sure this will work as intended? With ordering from overseas, its not too feasible ($) to send back the item..

The ebay item description states : "IBM ServeRAID M5016 SAS/SATA Controller for IBM System x3550 M3 Hersteller Art. Nr.: 90Y4304"

 

[mrkrad]

Why not get a L5639 dL180 g6 or DL380 ? The key part to this unit success is not the raid controller , yes that is important but the cost overall!

A DL180 G6 with dual L5639 $399 with rails - canot be beat on price. Throw enough ram to your solution and pick fast drives. 15K SAS 3.5" or 10K SAS 2.5" is key !

The raid controller with raid-10 will not be the limiting factor, it will be the drives. The cache you want to go with 512BBWC or 1024FBWC ( I suggest the FBWC) but the P410 is usually included in the HP solutions and works just fine with raid-10! Just need to ensure the BATTERY IS new or you get the super capacitor version (!!).

The M5014/m5015/m5016 are just LSI models, which are okay at best- but i'd never recommend them unless they were being used in a DELL or IBM server. The HP solution is far more robust and generationally the RISS is always compatible with HP product.

1. ESXi loves 10K SAS 2.5" or 15K SAS 3.5" - avoid near line 7200rpm drives (!!)

2. Battery back write cache/FBWC is a must for spinning disk! Ensure you get the best option possible!

3. LSI Battery is a PITA with its discharge cycle system and will destabilize an ESXi server when it shuts down the cache due to battery learning cycle.

4. The M5014/m5015 were problematic wit their batteries to say the least! I only use the M5014 because I am using it with SSD only. I do not use LSI controllers at all with disk product. They are imo CRAP unless you are talking ZFS/pass-through mode!

5. Do try to match the motherboard to the controller. IBM to IBM. I've run the M5014 in a dell, hp but honestly the UI is such crap and the drivers are so finicky i'd never recommend it to a friend to use on a daily basis.

6. HP rules. Sorry, the rest of the guys are junk. I've never experienced problems until the day I got my first 3 dell servers. PSOD in ESXi? Never saw that until my first R610 came on line with the Perc H700 firmware being bad for 6 months!

So there you go, if you must do LSI, get the retail version, with super capacitor. It alleviates the problems the oem's had with battery discharge! Their firmware still is crap but you can get a decent version of the firmware eventually.

I'd rather rock a P420/1gb FBWC ($269 new) on a HP DL180 G6 than throw a LSI 9260-8i but i've done both . Since the P410/P420 were never great with SSD, I had to rock what I had and that was the M5014 for $66 each. They worked great, and I have the extended feature keys $300 enabled and also LSI 9266-8i for the DL380 G6 - all running Samsung 840 pro 512gb ssd in raid-1 and esxi.

That is the only way i'd pick LSI over HP

I would never use near line 7200rpm drives with esxi! That is a kiss of death! I only have 1 VM with 12 x 2tb hitachi drives and i'm surprised it works still! RAID50 for veeam storage and fast retrieval. Surprised as heck that P420/1gb FBWC works with so many drives through an old extender (dl180 g6) but it works ! Very touchy server but solid as heck. Only 1 VM that is windows 2008R2 for disk to disk backup with veeam.

 

[sokolum]

The issue i having with the IBM M5015 on non IBM server hardware, is to upgrade the M5015 firmware. Still having this issue and can't no way (might be looking wrong) to upgrade the M5015 with IBM's own firmware. From here i don't experience issues.

 

[Rhinox]

...Rhinox, if ordering from this German seller gives you a brand new M5016 controller with the supercapacitor but what about the cache card? Can you confirm it was included with the controller?

Yes of course, cache is on the controller. It is a small PCB attached to the controller, lying slightly above the main controller PCB on the far side (opposite of bracket). Connector for attaching super-capacitor is on this cache daughter-board so it would not make sense to deliver this controller with super-cap and yet without cache (quite logical, without cache you do not need capacitor).

...Also, are there any compatibility issues with certain controller models and certain motherboards? How can I be sure this will work as intended?

You can never be absolutely sure, but I'd say this controller is compatible with any common PC-hardware as much as any other pci-express add-on card. It uses standard interface, has its own cache/cpu, actually it is small single-board single-purpose computer. You might find some exotic HW this controller is incompatible with, but I think with 99% of common stock HW not older than 3 years you should not have any problem. I tested this controller on 2 motherboards: Intel S1200BTL (small entry-level server board I plan to use for ESXi) and Asus P8Z68-V (my desktop). Both recognised this controller without problem and I could get into controller's bios. I wanted to try flashing new firmware, but it is already there. Maybe I will try to flash LSI-firmware (it is newer than IBM's)...

 

[Rhinox]

Why not get a L5639 dL180 g6 or DL380 ? The key part to this unit success is not the raid controller , yes that is important but the cost overall!

A DL180 G6 with dual L5639 $399 with rails - canot be beat on price. Throw enough ram to your solution and pick fast drives. 15K SAS 3.5" or 10K SAS 2.5" is key !

Well, I do not know where you are from but here (EU) DL180 starts at ~1100€ in configuration with E5504, 4GB RAM (24GB max), and simple raid0/1/10 (amazon). You still have to buy RAM, HDD, and better RAID-controller (that on-board one is sata2 and does not have cache). Now let's have a look at my home-built server:

Mobo Intel S1200BTRL 180€, Intel Xeon E3-1225v2 170€, 2U-case ~100€, PSU ~100€. That's ~550€ (all brand-new components). And I have higher-clocked better CPU (Ivy-bridge based). See the picture? I could build 2 servers for the price of single DL180. I did my math-homework before I started with it...

 

[PigLover]

Is that Amazon price new? I think the comparisons here are against used (off-lease). Look at eBay, not Amazon.

Of course if your business rules prevent you from considering used then your selections are not bad at all.

 

[lpallard]

Hey guys, thanks to all for replying! Great community I see here!

Let me answer everyone... Sorry for the infinite reply....

Thanks mrkrad for your detailled feedback! Much appreciated! Its a fresh & different line of thoughts from what I had in mind.

Why not get a L5639 dL180 g6 or DL380 ?

The idea behind this thread is to eliminate physical machines and not add new ones... Electricity bills and heat output are to blame... Noise is also to blame. I currently have a garage, but is accessible only from outside (Condominium unit) so for now noisy rackmount servers are out of the question.. When I change house, lets talk again

A DL180 G6 with dual L5639 $399 with rails - canot be beat on price.

I'd be amazed if you showed me where you saw that on fleabay... I just did a detailled search, sorting by price (low to high), in Canada (ebay.ca), nothing came up for less than $1,200... Most of them in the range of $1,800-$2,400. I did this search a few times between last summer and now, always in the thousands $$ so I concur with Rhinox on price observations.

15K SAS 3.5" or 10K SAS 2.5" is key !

Thanks for confirming SAS is the way to go. that clears up my mind about drive interface. While I am not so knowledgeable in the benefits of SAS over SATA for a virtualization machine, I understand the benefits of reliability and data integrity that SAS brings over SATA are essential.

The raid controller with raid-10 will not be the limiting factor, it will be the drives.

Let me ask this: for RAID10, you would need 4 driuves correct? How can you assemble a 4 drive array on a controller with only 2 SAS ports? Split cables? Then if accurate, doesnt' it impede the bandwidth which is the whole point of raid 0 at first?

The M5014/m5015/m5016 are just LSI models, which are okay at best- but i'd never recommend them unless they were being used in a DELL or IBM server.

HP rules. Sorry, the rest of the guys are junk.

I do not use LSI controllers at all with disk product. They are imo CRAP unless you are talking ZFS/pass-through mode!

Reading this comment, I feel that you do not recommend LSI (hence the M5016) products. Is it because of hardware quality? Firmware? Lack of features versus the competition? All of that? What does "crap/junk" means?

avoid near line 7200rpm drives (!!)

I would never use near line 7200rpm drives with esxi! That is a kiss of death!

Good ! I was going to use nearline 7200RPM drives at first until I saw those comments!!!

Battery back write cache/FBWC is a must for spinning disk! Ensure you get the best option possible!

OK Good for the supercap or BBU and cache.... I know what to do now!

LSI Battery is a PITA with its discharge cycle ....

Also applicable for the M5016's supercap?

Do try to match the motherboard to the controller. IBM to IBM.

That would confirm my worst fears: I order the M5016 (or any RAID controller for that matter), plug it on my supermicro H8DCL-IF mobo and its a no show....

The issue i having with the IBM M5015 on non IBM server hardware, is to upgrade the M5015 firmware. Still having this issue and can't no way (might be looking wrong) to upgrade the M5015 with IBM's own firmware. From here i don't experience issues.

Sokolum, so if I understand you correctly, you are trying to flash your M5015 to the latest FW, and it doesnt work, and you assume its because you are on a non-IBM platform? Is that correct? Also, what do you mean by "From here i don't experience issues."?? Do you mean using the M5016 on the Supermicro H8DCL-IF, i.e. my specific setup?

You can never be absolutely sure, but I'd say this controller is compatible with any common PC-hardware as much as any other pci-express add-on card. It uses standard interface, has its own cache/cpu, actually it is small single-board single-purpose computer. You might find some exotic HW this controller is incompatible with, but I think with 99% of common stock HW not older than 3 years you should not have any problem. I tested this controller on 2 motherboards: Intel S1200BTL (small entry-level server board I plan to use for ESXi) and Asus P8Z68-V (my desktop). Both recognised this controller without problem and I could get into controller's bios. I wanted to try flashing new firmware, but it is already there. Maybe I will try to flash LSI-firmware (it is newer than IBM's)...

Rhinox, thats good feedback to hear. Perhaps contacting Supermicro before I do purchase would be a way to confirm beforehand that it will work? They (SM) have been excellent so far with me on other support topics... Quick and accurate "I am taking you by the hand" type of support.

Anyways, we have a good discussion going on here, and I am learning a great deal too so I think this is the way to go!!

I am curious to see what others and you too have to say!

Thanks

 

[Jeggs101]

Award for most quotes in a single post goes to you my friend

 

[lpallard]

Award for most quotes in a single post goes to you my friend

Thanks! And Im quoting you on that!

 

[Jeggs101]

Thanks! And Im quoting you on that!

Good form! Well played.