Raid Card recommendation for ESXi/Proxmox

[JohnnyBeGood]

Hi all,

I got Lenovo ThinkServer TS140 and now I'm thinking to get RAID controller because from what I've read ESXi does not support software RAID that TS140 has.
Right now I'm not sure if I will settle with ESXi or Proxmox I guess I will have to test each one and decide.
Can someone suggest RAID card? LSI MegaRAID 9260-8i seems to be trusted.

TIA

 

[Patrick]

What drives are you going to use?

Single node or multiple nodes?

Do you need pass-through?

Proxmox can have ZFS installed as standard so likely you would just use that or md raid (since it is a Debian based system.) If you had 3-4 or more nodes, then Ceph may make more sense.

If you need a base datastore for ESXi, then it would just depend on drives you wanted to use.

 

[JohnnyBeGood]

Forgot to mention that starting to explore this field, so please go easy on me
I'm trying to decide between either 3GB or 4GB WD Red.
This is for home use. So I guess single node?
Not sure what pass-through is and how it relates to virtulaization?

 

[MiniKnight]

We've all started somewhere.

If this is for a job, learn ESXi (probably). ESXi has more $$$ behind it from corporations and therefore a perpetual buzz.

KVM and LXC (what Proxmox uses) are very popular as well but there's not as good of a certification track. Most of the newer converged appliance guys and OpenStack guys are KVM so good to learn too.

If it is only a drive or two, I'd start with Proxmox or maybe better is just installing CentOS or Ubuntu. You'll learn a lot and it'll save you like $100 on the IBM M1015 or more on a better card.

Basically get a LSI 9211-8i or equivalent (M1015) like this: LSI Internal SATA/SAS PCI-e RAID Controller Card SAS9211-8i 8 PORT HBA if you think you'll want 5-8 hard drives. If just 1-4 LSI SATA/SAS 9211-4i 6Gbps PCI-Express 2.0 will be OK. Those are not real RAID cards but will let your SATA devices pass through if you wanted to make a storage VM (like a ZFS all-in-one.) They also handle RAID 0 and RAID 1 no problem.

If you want the full RAID card experience: LSI 9260-8i 512MB PCIe 8 Port SAS SATA 6Gb/s LP MegaRAID Controller LSI00202 that'll do higher level parity RAID like RAID 5.

 

[Keljian]

Basically get a LSI 9211-8i or equivalent (M1015) like this: LSI Internal SATA/SAS PCI-e RAID Controller Card SAS9211-8i 8 PORT HBA if you think you'll want 5-8 hard drives. If just 1-4 LSI SATA/SAS 9211-4i 6Gbps PCI-Express 2.0 will be OK. Those are not real RAID cards but will let your SATA devices pass through if you wanted to make a storage VM (like a ZFS all-in-one.) They also handle RAID 0 and RAID 1 no problem.

If you want the full RAID card experience: LSI 9260-8i 512MB PCIe 8 Port SAS SATA 6Gb/s LP MegaRAID Controller LSI00202 that'll do higher level parity RAID like RAID 5.

This - Definitely this.

 

[JohnnyBeGood]

We've all started somewhere.

If this is for a job, learn ESXi (probably). ESXi has more $$$ behind it from corporations and therefore a perpetual buzz.

KVM and LXC (what Proxmox uses) are very popular as well but there's not as good of a certification track. Most of the newer converged appliance guys and OpenStack guys are KVM so good to learn too.

If it is only a drive or two, I'd start with Proxmox or maybe better is just installing CentOS or Ubuntu. You'll learn a lot and it'll save you like $100 on the IBM M1015 or more on a better card.

Basically get a LSI 9211-8i or equivalent (M1015) like this: LSI Internal SATA/SAS PCI-e RAID Controller Card SAS9211-8i 8 PORT HBA if you think you'll want 5-8 hard drives. If just 1-4 LSI SATA/SAS 9211-4i 6Gbps PCI-Express 2.0 will be OK. Those are not real RAID cards but will let your SATA devices pass through if you wanted to make a storage VM (like a ZFS all-in-one.) They also handle RAID 0 and RAID 1 no problem.

If you want the full RAID card experience: LSI 9260-8i 512MB PCIe 8 Port SAS SATA 6Gb/s LP MegaRAID Controller LSI00202 that'll do higher level parity RAID like RAID 5.

Thanks for taking time to explain and for links especially.

My initial goal is to setup FreeNAS with ZFS (or anything else that will be safe even after one drive dies) for in home storage of surveillance family videos and pictures and second learn about virtualization.

I do want full RAID experience so I will go with LSI 9260-8i I just need to find one that comes with battery ie. this one LSI LSI-25121-74B 9260-8i 6Gb/s SATA+SAS MegaRaid Raid Controller w/ Cable & BBU

 

[EluRex]

If you want to run a single with ZFS, then it is better for you to use proxmox ve4 as as esxi does not support zfs natively and its vSAN has many limitation and if my memory serve my correctly, only nexenta platform support vSAN zfs for esxi. But that will be a two nodes configuration.

if you want to set up FreeNAS with ZFS, you have two options:

(1) have pci-passthrough letting zfs handling the software raid/hard drive directly, zfs does not like any software middle layer btw the sata/raid controller and hd.

(2) install pve 4.0 on ZFS pool as root filesystem, and mount your zfs pool to a freebsd container. That way your pve handles the zfs, and freenas handles the file server permissions and web interface etc

 

[JohnnyBeGood]

If you want to run a single with ZFS, then it is better for you to use proxmox ve4 as as esxi does not support zfs natively and its vSAN has many limitation and if my memory serve my correctly, only nexenta platform support vSAN zfs for esxi. But that will be a two nodes configuration.

if you want to set up FreeNAS with ZFS, you have two options:

(1) have pci-passthrough letting zfs handling the software raid/hard drive directly, zfs does not like any software middle layer btw the sata/raid controller and hd.

(2) install pve 4.0 on ZFS pool as root filesystem, and mount your zfs pool to a freebsd container. That way your pve handles the zfs, and freenas handles the file server permissions and web interface etc

I keep coming across pci-passthrough but I'm not sure what it does. Here's one explanation Pci passthrough - Proxmox VE
"PCI passthrough allows you to use a physical PCI device (graphic card, network card) inside a VM (KVM virtualization only) If you "PCI passthrough" a device, the device is not available in the host anymore."
if I understabd it correctly dedicated entire ie. graphics card to the ProxMox and if I wanted to connect ie. monitor to my server I would not see anything, am I wrong?

I guess option 2 is the best but again I'm new to this and willing to learn. When you say "install pve 4.0 on ZFS pool as root filesystem" where would I create that ZFS pool?
Best would be to start experimenting before putting it online.

 

[EluRex]

PCI passthrough basically means instead the host utilize the pci device, it passes the pci device to a VM to use it directly, so that the VM can use the hardware acceleration function or other functions in the hardware device.

Most common use for pci passthrough is vga pass through to let a VM can have vga hardware acceleration/decoding. Or a sata controller to a storage VM Node to handle HDs/Software RAID etc

for option 2, you may follow http://www.servethehome.com/proxmox-ve-3-4-zfs-raid-1-boot-disks/

 

[canta]

I sugget to use real raidcard 9260 OEM .
TS140 has max memory 32G
if you use ZoL in proxmox. you have to know zfs loves memory and need to limit ~8G in ts140 that will be used by zfs.

my solution was, using m5014 (already have) and set to RAID 1 , since memory is precious on ts140 ..

do not use m1015 or other OEM for raid 1, they slow as hell!! since not having cache memory/flash.

esxi -> use raid card 9260 OEM for sure, no choice due on esx does not support software raid.

 

[JohnnyBeGood]

I did more search on IBM ServeRAID Mxxxx line and came across ServeRAID M5016 IBM ServeRAID M5016 Controller Raid SAS/SATA PCIe 6GB 1G cache =LSI 9265-8i and that one does not have a battery back but super-capacitor instead. Sounded cool but when I asked the seller if it came with it he referred me to this BBU09 for LSI 49571-03 9286 9286CV-8e 9285 9285CV-8e IBM M5110 M5016 that's a BBU? for some reason I was think it was more like a flash card chip.
So I'm totally lost. I really like the idea not having to worry about battery but when I add that to the price of the card its going to be over $200.
Are there any other options?

 

[Keljian]

If you are using freenas, then there is absolutely positively no point going a raid card with a BBU as you are just going to want to pass the drives to the operating system, not create raid using raid functions on the card.

Get a 9211-8i or 9240-8i based card like the m1015 and flash it in IT mode, get a couple of 8087 cables that terminate in SATA, and be done with it. If you want to make sure your data is safe beyond this, add a UPS

 

[canta]

If you are using freenas, then there is absolutely positively no point going a raid card with a BBU as you are just going to want to pass the drives to the operating system, not create raid using raid functions on the card.

Get a 9211-8i or 9240-8i based card like the m1015 and flash it in IT mode, get a couple of 8087 cables that terminate in SATA, and be done with it. If you want to make sure your data is safe beyond this, add a UPS

this is depends on your build
since e3/i3 is limited to 32G ram...
if you used ZFS and you need to set minimally 8G(or less that depend on yours) ram max that use by ZFS if needed.

This is the reason I pick real raid for my proxmox. and use only raid 10 (I think, totally forgot, need to check) for my proxmox os and proxmox VM.


UPS is a must . try to get UPS that has remote management and extra features (load consumption, ssh, API to access, snmp( I do not use actually, some loves to use snmp), email notification, and others, set min and max over/undershoot voltage, temp sensors, or more).
used APC (smart UPS) or other well know brand would be a good bet on ebay .

 

[Alex Skysilk]

All good posts, but its hard to make suggestion/performance recommendations since OP did not specify what the application is.

You have a TS140 with either a dual or quad core CPU, with somwhere between 4-16GB of ram. Based on the discussion, you want to:

1. Use it as a bare metal hypervisor. Options discussed are Vmware ESXi and Proxmox, and I'll add Microsoft Hyper-V as well. You would want to use a VMWare supported raid controller if your intention is to use ESXi; for Proxmox and Hyper-V its not necessary and you can/should use a SAS passthrough card such as the aforementioned IBM M1015, as you'll want to use their native LVM functionality.
2. use it as a storage server. Options discussed are freenas (I'll add Nas4Free as a lighter alternative.) For either one of those you can/should use a SAS passthrough card such as the aforementioned IBM M1015, as you'll want to use their native LVM functionality.
3. you want to use a hyperconverged (storage, hypervisor) deployment such as Openstack. NOT RECOMMENDED FOR YOUR USE CASE- without a cluster you gain nothing over either of the above, and your hardware is too light in resources to run it effectively; however, if you wanted to set it up in preparation for adding more nodes, Proxmox/Ceph is probably the best approach from a cost/complexity standpoint. see option 1.

tl;dr- you want an IBM M1015.

Good luck

 

[Keljian]

Thanks for taking time to explain and for links especially.

My initial goal is to setup FreeNAS with ZFS (or anything else that will be safe even after one drive dies) for in home storage of surveillance family videos and pictures and second learn about virtualization.

[/URL]

Pretty clear what he wants. To me the answer is clear cut, get a Xeon based board with ecc, or applicable server, get a Lsi 9211 or 9240, plus cables, implement. If you want to get fancy, go 10 gbit for the network.

 

[kroem]

Adding to the confusion here, but: I see it like Op would want hw-raid for datastore (9260/M5014/5/6) and then a M1015 equivalent to passthrough to a storage VM.

Or live le dangerous life and only backup datastore files every now and then...(ie No hw-raid)

 

[Keljian]

Raid is not a backup solution...

 

[MrCalvin]

My experience is the ESX is NOT a good choice if you don't have RAID controller with cache RAM! (which is very expensive).
Unless using SSD you easily run into BIG storage performance issues.
Sure, you can easily transfer smaller files, but try to copy a 10GB file to you guest OS and you will se a nightmare!!!
Hyper-V handle this mush better, haven't tested Proxmox VE. Another strange thing about ESX, it does'nt seem to support newer HDDs with 4K sectors sizes, again Hyper-V does, don't know about Proxmox VE.?

 

[gea]

ESXi is a type-1 Hypervisor.
Greatest advantage is the smallest footprint, its more like a firmware with Web-management (Use ESXi 6.5d). This means it is quite the fastest option with the lowest RAM needs, the fastest to setup and restore and with best support for any type of OSes including non Linux guests. ESXi offers full virtualisation. For Linux container support you can add a Linux guest with docker support.

Regarding storage ESXi is very limited. VMware offers vsan but usually you combine ESXi with a dedicated storage server either as a separate iSCSI/NFS server or as a virtualized storage appliance.

The last option is one that I use for years. I also offer a ready to use ZFS storage appliance with a HowTo. In such a combination you can use ESXi with all storage features of a highend storage SAN server.

see my HowTo
http://napp-it.org/doc/downloads/napp-in-one.pdf

 

[JohnnyBeGood]

Thanks guys!