Need advice for a lab refresh, not sure which direction to go in

[idea]

I am seeking advice on what I should do. I currently have one box (check my sig). It is set up in a "all in one box solution" which would never be found in a datacenter, but its great for a home lab for obvious reasons. It's been running solid for a while. I'm just bored and ready to try something new. I also want to try more advanced features of vSphere using multiple ESX hosts.

My requirements:

• I want to learn vSphere's more advanced features so I will need 2+ bare metal ESX hosts
• Shared storage. I will be using Linux + ZFS on Linux (aka "ZoL") now that it is mature enough
• For general storage, I currently have 6x2TB RAIDZ2 (RAID6), I don't think I would need any more than that
• For fast storage, I currently have 4x 10K 146gb SAS drives in a striped ZFS mirror, I don't think I would need any more than that
• NO hardware RAID controllers. All data, including VM guests, will reside on ZFS storage
• Rackmountable chassis only
• Cool, quiet, and power efficient
• If I can't have perfectly cool, quiet, and power efficient then I must come as close as I possibly can to this goal
• Capacity for comfortably running 10-20 guests with very low load
• At least 16GB per ESX host
• Nice cable management, no hacked up parts, has to look sexy in my new 25U cabinet

I can think of four possible ways to go about this:

#1 - Two ESX whiteboxes plus shared storage using 10gig ethernet (3 physical boxes)
Build two physical diskless ESX hosts with 10Gbe NIC, and connect them to a storage box. Boot them with USB disks
Repurpose box in my signature for file server only, install a dual 10Gbe NIC and connect directly to each ESX host
Each box will be based off E3-1230 Xeon, Supermicro X9SCM, 16 or 32GB RAM

#2 - Dell C6100 with 3 ESX hosts and 1 fileserver for shared storage among them
2U form factor vs. at least 4-6U, maybe even 12U with the above setup
Cheaper than the above setup
I don't care about older generation Intel CPU tech
I'm definitely going to need to do PigLover's fan mod
Each node will get an QDR Infiniband mezz card, and I will buy a DDR switch (cheaper than QDR) and the cables to network it all together

#3 - One single Dell C6100 to do it all! Two ESX hosts, and two fileservers for shared storage. (1 fileserver for slow "nearline" storage, and 1 fileserver for fast SAS storage)
Because the maximum number of disk bays you can assign to a node is 6, and I need to make use of all 12, I would do the following:
Node #1- Fileserver #1 - Connect 6x 3.5" 2TB SATA, create a RAIDZ2 storage pool and share via NFS/iSCSI as needed
Node #2- Fileserver #2 - Connect 6x 3.5" 146GB SAS disks, create a 3-way ZFS mirror and share via NFS/iSCSI as needed to ESX hosts and database servers
Node #3- ESX host
Node #4- ESX host
Each node will get an QDR Infiniband mezz card, and I will buy a DDR switch (cheaper than QDR) and the cables to network it all together

#4 - Diskless Dell C6100 with 3-4 ESX hosts and separate physical fileserver (repurpose the box in my sig)
C6100 nodes #1-4 will be ESX hosts, or maybe I'll leave 1 for a pfSense system
C6100 nodes will boot off USB drives
The box in my sig will be repurposed as a fileserver and provide SATA/SAS storage via NFS/iSCSI
Each node will get an QDR Infiniband mezz card, and I will buy a DDR switch (cheaper than QDR) and the cables to network it all together
* Also need a QDR Infiniband PCI-E card for the separate fileserver

What would you do if you were me?

 

[idea]

edited: I added option 4, and I didn't realize my sig wasn't showing

 

[idea]

I've read up on Infiniband so I decided to edit the post, including details where necessary

Basically I will be fitting QDR Infiniband mezz cards to each node accessing the SAN/serving the SAN
Also buying a DDR switch and cables. DDR switches are about ~$200 and QDR switches are about $1000. I don't need the speed, I'd rather pocket the difference
If I do a separate physical fileserver I will be needing a PCI-E Infiniband adapter as well

 

[idea]

Damn it, the C6100 has a crappy onboard SATA controller. If I choose to use one (or two) node/s as a fileserver then I need the 1068E mezz card to utilize SAS disks. And ZFS loves the 1068E. I can't have both the 1068E and QDR mezz cards though. Maybe a separate fileserver is the best way to go...

 

[Fzdog2]

I personally run like your option #4 and like the setup. The only difference is that I run my fileserver(FreeNAS) behind ESXi on my Supermicro 24bay. This lets me consolidate all the VM's onto that box during idle/low load times and keeps the C6100 powered off most of the time. And with vCenter DRS and DPM, if the CPU or memory load gets to be too much for the 1 node, it will power up C6100 nodes to balance it out. Though I have found that my 3x146GB RaidZ1 puts out next to nothing for IOPS so I am looking for a better solution to give me better speed for the NFS shared storage.

 

[PigLover]

I personally run like your option #4 and like the setup. The only difference is that I run my fileserver(FreeNAS) behind ESXi on my Supermicro 24bay. This lets me consolidate all the VM's onto that box during idle/low load times and keeps the C6100 powered off most of the time. And with vCenter DRS and DPM, if the CPU or memory load gets to be too much for the 1 node, it will power up C6100 nodes to balance it out. Though I have found that my 3x146GB RaidZ1 puts out next to nothing for IOPS so I am looking for a better solution to give me better speed for the NFS shared storage.

+1 for this. Fabulous, except that you will still need something to act as backup fileserver. Being able to consolidate onto one server is very handy not only in low-use times but for maintenance/upgrade/changes to the rest of the system. Replace the 4x 146GB SAS with 4x 200+ GB SSD and your golden for IOPs (ok - lots of money - sorry, but it is the easy fix to that problem).

Might want to boot ESXi in the C6100 nodes off a single disk rather than USB. Shoving one disk in front for each node is no big deal, but there is no good way to use USB boot unless you hang it out the back (dangerous - will get knocked off) or do some mods. Use whatever disks are handy, old SATA drives, cheap laptop drives or old/small SSDs if you have them. It only needs to boot ESXi. If you do my fan mod you should plug the drive nto the middle drive slot for each node so that you have unobstructed airflow below for nodes 1 & 2 and above for nodes 2 & 4. The slower/quieter fans are just a little uinderpowered to keep it all cool if you have the drive slots all full.

This is very similar to where my rack is heading, except I am using Proxmox instead of ESXi for the VM management and I am moving towards ZoL loaded onto the Proxmox node instead of running a NAS on a VM - but basic concept is almost identical. And I've built a 14-drive backup NAS on an HP DL180, also running Proxmox and available as an additional VM node if necessary.

 

[britinpdx]

I can't have both the 1068E and QDR mezz cards though.

True, but you could consider using the mezz slot for one of the functions and the pci-e slot for the other.

Here's one of my sleds configured with the Infiniband mezz card and an IBM M1015 providing the SAS/SATA capability. I'm simply using the M1015 to provide a higher bandwidth to SSD's than the onboard SATA.

The M1015 is not too long so as to interfere with cpu1 heatsink. The Monoprice SSF-8087 to SATA cable is carefully routed in-between the 5520 chipset heatsink and the mezz card and down the opposite side of the sled from the normal SATA routing. It's potentially reducing airflow in a critical area, but so far seems to be OK.






Roles could be reversed, with the LSI mezz card and and infiniband card in the pci-e slot. I've several Mellanox pci-e Infiniband cards but not all of them are short enough so as not to interfere with CPU1 heatsink.

 

[Fzdog2]

+1 for this. Fabulous, except that you will still need something to act as backup fileserver. Being able to consolidate onto one server is very handy not only in low-use times but for maintenance/upgrade/changes to the rest of the system. Replace the 4x 146GB SAS with 4x 200+ GB SSD and your golden for IOPs (ok - lots of money - sorry, but it is the easy fix to that problem).

Might want to boot ESXi in the C6100 nodes off a single disk rather than USB. Shoving one disk in front for each node is no big deal, but there is no good way to use USB boot unless you hang it out the back (dangerous - will get knocked off) or do some mods. Use whatever disks are handy, old SATA drives, cheap laptop drives or old/small SSDs if you have them. It only needs to boot ESXi. If you do my fan mod you should plug the drive nto the middle drive slot for each node so that you have unobstructed airflow below for nodes 1 & 2 and above for nodes 2 & 4. The slower/quieter fans are just a little uinderpowered to keep it all cool if you have the drive slots all full.

Unfortunately everytime I buy another SSD, I end up using it as a boot drive replacement for one of my computers at home. I've now got a collection of 5 various SSD's and only 1 of them is used in my servers.

For the ESXi boot stuff, I need to sit down for a few hours and finish the vCenter Auto Deploy setup I began. Then I can just boot off a single file share image over iSCSI and forget any physical media.

 

[idea]

I personally run like your option #4 and like the setup. The only difference is that I run my fileserver(FreeNAS) behind ESXi on my Supermicro 24bay. This lets me consolidate all the VM's onto that box during idle/low load times and keeps the C6100 powered off most of the time. And with vCenter DRS and DPM, if the CPU or memory load gets to be too much for the 1 node, it will power up C6100 nodes to balance it out. Though I have found that my 3x146GB RaidZ1 puts out next to nothing for IOPS so I am looking for a better solution to give me better speed for the NFS shared storage.

That sounds exactly like how I wrote out #4 (separate storage system behind ESX) but I'm confused... how does that allow you to "consolidate all the VM's onto that box during idle/low load times" and how does your C6100 stay powered off most of the time?

I can't wait to try out vCenter DRS and DPM. Features like that are the reason I started this project.

And I'm sure you already know this but your 3x146GB RAIDZ, just like RAID5, has the IOPS of a SINGLE disk. So yeah, that's a problem. You could add one more 146GB and rebuild your zpool into a RAID10-like array with "zpool create tank mirror disk1 disk2 mirror disk3 disk4" which coincidentally is how I have it

 

[Fzdog2]

That sounds exactly like how I wrote out #4 (separate storage system behind ESX) but I'm confused... how does that allow you to "consolidate all the VM's onto that box during idle/low load times" and how does your C6100 stay powered off most of the time?

I can't wait to try out vCenter DRS and DPM. Features like that are the reason I started this project.

And I'm sure you already know this but your 3x146GB RAIDZ, just like RAID5, has the IOPS of a SINGLE disk. So yeah, that's a problem. You could add one more 146GB and rebuild your zpool into a RAID10-like array with "zpool create tank mirror disk1 disk2 mirror disk3 disk4" which coincidentally is how I have it

My Supermicro chassis has ESXi installed to onboard USB drive, and then I have 2 FreeNAS VM's that share out storage back through NFS to be used by all ESXi hosts as shared storage to enable VMWare HA/DRS/DPM etc.
I have DRS set to never power off my main storage chassis so when the VM load is low enough to be ran off of 1 server (which is 99% of the time), all the VM's are kept on that node and DPM turns all 4 of the C6100 nodes off. If that one server is deemed unable to provide adequate resources, DPM will send the power up command over IPMI to boot up a C6100 node. When that server gets seen in vCenter, DRS will balance out the VM's as it sees fit.

For the storage, the only thing I can do for the short term is to put a spare 32GB SSD in as a ZIL device on the RAIDZ1.

Edit: I get it now I see you have an AIO setup.

 

[idea]

+1 for this. Fabulous, except that you will still need something to act as backup fileserver.

I actually have a backup fileserver, so how would that work into this situation?

Might want to boot ESXi in the C6100 nodes off a single disk rather than USB. Shoving one disk in front for each node is no big deal, but there is no good way to use USB boot unless you hang it out the back (dangerous - will get knocked off) or do some mods. Use whatever disks are handy, old SATA drives, cheap laptop drives or old/small SSDs if you have them. It only needs to boot ESXi. If you do my fan mod you should plug the drive nto the middle drive slot for each node so that you have unobstructed airflow below for nodes 1 & 2 and above for nodes 2 & 4. The slower/quieter fans are just a little uinderpowered to keep it all cool if you have the drive slots all full.

If that is the only downside to booting from USB then I don't mind. I have two really tiny USB drives that are impossible to knock out

 

[idea]

True, but you could consider using the mezz slot for one of the functions and the pci-e slot for the other.

Here's one of my sleds configured with the Infiniband mezz card and an IBM M1015 providing the SAS/SATA capability. I'm simply using the M1015 to provide a higher bandwidth to SSD's than the onboard SATA.

The M1015 is not too long so as to interfere with cpu1 heatsink. The Monoprice SSF-8087 to SATA cable is carefully routed in-between the 5520 chipset heatsink and the mezz card and down the opposite side of the sled from the normal SATA routing. It's potentially reducing airflow in a critical area, but so far seems to be OK.

Roles could be reversed, with the LSI mezz card and and infiniband card in the pci-e slot. I've several Mellanox pci-e Infiniband cards but not all of them are short enough so as not to interfere with CPU1 heatsink.

That is awesome info! This puts option #3 back in the running, which means pack everything (hypervisors AND storage) into one C6100. Great pics too. You a photographer or something?

Question: What is your limiting you from connecting more than 4 drives (I only see one breakout cable connected. Also, do drive LEDs work? Any other annoyances?

 

[cafcwest]

True, but you could consider using the mezz slot for one of the functions and the pci-e slot for the other.

Here's one of my sleds configured with the Infiniband mezz card and an IBM M1015 providing the SAS/SATA capability. I'm simply using the M1015 to provide a higher bandwidth to SSD's than the onboard SATA.

Would love to hear a bit more about what you have set up here - specifically, drives and performance. Backplanes support SATA 6Gb?

 

[idea]

My Supermicro chassis has ESXi installed to onboard USB drive, and then I have 2 FreeNAS VM's that share out storage back through NFS to be used by all ESXi hosts as shared storage to enable VMWare HA/DRS/DPM etc.
I have DRS set to never power off my main storage chassis so when the VM load is low enough to be ran off of 1 server (which is 99% of the time), all the VM's are kept on that node and DPM turns all 4 of the C6100 nodes off. If that one server is deemed unable to provide adequate resources, DPM will send the power up command over IPMI to boot up a C6100 node. When that server gets seen in vCenter, DRS will balance out the VM's as it sees fit.

For the storage, the only thing I can do for the short term is to put a spare 32GB SSD in as a ZIL device on the RAIDZ1.

Edit: I get it now I see you have an AIO setup.

Wow I get it. Your separate fileserver is also an AIO setup just like mine, plus it is using a buncha fancy VMware abbreviated stuff to balance the load to your C6100 when needed. That is a very cool and very interesting set up, especially with the IPMI boot up trick. Does it power down also?

Would you happen to have any documentation on this sort of setup? I really like your idea because I can do the same exact thing as you just by adding a C6100 to my lab since I already have the AIO box. Problem is, I kind of made a decision to migrate off of AIO because it's kind of an unconventional setup and doesn't mimic a real world production scenario. But you're making me feel guilty for having the storage server AND the C6100 powered on at all times

 

[Biren78]

Would love to hear a bit more about what you have set up here - specifically, drives and performance. Backplanes support SATA 6Gb?

Can I second this request?

 

[idea]

Option #3 has a huge problem. If I use all 12 disks for RAID arrays, then I have no disks left to boot from. That sucks. Two of my nodes will be file servers (with 6 drives each) and since I won't settle for anything less than a mirrored pair of boot disks for each of them, then I need 4 devices for booting alone. The other 2 nodes will be ESX which is USB-drive-bootable-friendly and I don't care if it fails, the second hypervisor will continue on until I can replace the USB drive

Here is the proposed setup once again:
One C6100 with SAS 1068E mezz cards (in the future I can replace them with LSI SAS2008 PCI-E cards if I need the mezz slot, or if I want 3TB+ drives, etc)
-Node #1 w/ 6 disks - Linux serving up ZFS with (6x) 2TB RAIDZ2 (raid6)
-Node #2 w/ 6 disks - Linux serving up ZFS with (6x) 146GB 10K SAS, for 3x mirrors striped together
-Node #3-4 ESX Hosts

I truly love this idea because I can consolidate EVERYTHING into one C6100 but perhaps it's just too much to ask of one single chassis to handle. Looks like I'll be having an additional fileserver. 4U more space, more power, more heat, more noise.... damn why did I fall in love with #3

 

[idea]

Option #3 has a huge problem. If I use all 12 disks for RAID arrays, then I have no disks left to boot from. That sucks. Two of my nodes will be file servers (with 6 drives each) and since I won't settle for anything less than a mirrored pair of boot disks for each of them, then I need 4 devices for booting alone. The other 2 nodes will be ESX which is USB-drive-bootable-friendly and I don't care if it fails, the second hypervisor will continue on until I can replace the USB drive

Here is the proposed setup once again:
One C6100 with SAS 1068E mezz cards (in the future I can replace them with LSI SAS2008 PCI-E cards if I need the mezz slot, or if I want 3TB+ drives, etc)
-Node #1 w/ 6 disks - Linux serving up ZFS with (6x) 2TB RAIDZ2 (raid6)
-Node #2 w/ 6 disks - Linux serving up ZFS with (6x) 146GB 10K SAS, for 3x mirrors striped together
-Node #3-4 ESX Hosts

I truly love this idea because I can consolidate EVERYTHING into one C6100 but perhaps it's just too much to ask of one single chassis to handle. Looks like I'll be having an additional fileserver. 4U more space, more power, more heat, more noise.... damn why did I fall in love with #3

Ok, the #3 idea just won't die. It has one hope left: If I use the PCI-E slot for a SAS2008-based card (i.e. IBM M1015) just like britinpdx did, I will get 8 ports, all of which will be routed to the front of the chassis and broken out into 8 SATA ports. 6 of them go to the backplane. 2 are left over. I wonder if I can stuff two SATA disks up front somewhere? Maybe two 1.8" drives or a tiny SATA DOM? They will also require power from somewhere. Maybe #3 just hopeless. I'll be looking into it

 

[idea]

Maybe this will work but I need help from someone who has a C6100 tray handy. Can you please tell me if there is enough clearance to fit a PCI card with 1.5" of height? This device will give me an additional two 2.5" drives which are connected through the PCI slot so they require no SATA or power cables.

SYBAUSA - SD-PEX40044 >> RAID Caddy for Two 2.5" SATA 6Gbps Hard Disk Drives, PCI-Express Card, Marvell 88SE9128 Chipset

 

[dba]

Maybe this will work but I need help from someone who has a C6100 tray handy. Can you please tell me if there is enough clearance to fit a PCI card with 1.5" of height?

Sorry, that PCIe card will definitely not fit.

 

[Fzdog2]

Wow I get it. Your separate fileserver is also an AIO setup just like mine, plus it is using a buncha fancy VMware abbreviated stuff to balance the load to your C6100 when needed. That is a very cool and very interesting set up, especially with the IPMI boot up trick. Does it power down also?

Would you happen to have any documentation on this sort of setup? I really like your idea because I can do the same exact thing as you just by adding a C6100 to my lab since I already have the AIO box. Problem is, I kind of made a decision to migrate off of AIO because it's kind of an unconventional setup and doesn't mimic a real world production scenario. But you're making me feel guilty for having the storage server AND the C6100 powered on at all times

Yes it auto powers down nodes if they are deemed to be unneeded for the current workload.

Eventually I will update my build thread with my current setup, just haven't had the time yet.