vmware + storage box or just HyperV it?

[KnottyMan]

CAD Company I'm supporting started with 3 employees, owner included, and a server that I built for them. Ubuntu 14.04, 18TB raid 6 on an Areca 1224 with WD 3TB reds. Everything has been working great for three years. Built the CAD workstations too. Latest are i7 6850K with 128GB memory, GTX 1080, fast fast fast is the order of the day. Drawings can be 10MB each with 12 other xref drawings attached so loading takes a bit. And sometimes they have two sets of these open at a time to compare, etc. RAM++!

Now they're 20 users though, and we're moving offices. Business is well established so let's spend some money for a core upgrade. I'm looking at Supermicro 1028U-TN10RT+ or 2028U-TN24R4T+. Leaning towards the 2U for storage expandability. They're all flash, Mellanox dual 10Gb, etc. But good night they're expensive. 6x 2TB P3520s will do that I guess. They're using 2.5TB actively right now. Need file server expansion plus other VM storage.

One of my current plans is to have a separate box for vmware that connects to the supermicro via iscsi. Run Freenas with raidZ2 on the SM, all else on vmware. Make new VMs for AD, Windows file server, OPNsense, etc. I know, upside down triangle.

Or, do I just say screw it and have one bad ass HyperV machine with storage spaces? (bleh) I'm much more comfortable with vmware. Maybe 2k16 is better. I'm new to Freenas but I have it successfully working in my home stack on dual 10G. I know Freenas can do SMB, I think I'd rather do it via Windows VM. I could do a chicken in the egg thing with a single box, Freenas VM it and pass the SSDs, but that skeeves me out too.

Or I could build a new box with a Dell R510 12 bay with WD REs or 7K Ultrastors and an Areca 1883 or 9211 and save a ton of money, ie, is all this nvme flash going to really do any good since all the workstations are 1GbE (but a lot of them) anyway? Supermicro 2U is clocking in at $14k, plus I need to charge them sales tax (8.7%) and make a little money myself would be nice.

 

[Tom5051]

Why not use ESXi with RAID passthrough to the file server VM?
or local RDMs?
Not a VMware supported config for a production environment but it works.

 

[KnottyMan]

I could pass SSDs to the VM, but then what do I do for the others? With only 6 disks, I could hold one back, but then I just have one disk for all other VM IO. It would be nice to have vsan/storage spaces/amalgamationwhatever, but for just one host that I can manage from vmware. Which is why I ask if I should just cave and go one box HV...

I've done areca raid as the ESXi datastore, and that works too, but there's no management. So if a drive fails, gotta be present at the box and reboot. Can't do IPMI/KVM since the virt VPN server is down...

 

[Rand__]

For Vsan you will at least need 3 hosts (Robo+Witness) unless you go virtual.

The question is - does it make sense to build a single point of failure? How soon do they really suffer if that one-box is gone? One week is unlikely but not impossible (think illness etc).

So If I were you I'd strongly consider to build something more reliable here, at least two hosts with appropriate redundancy. Vsan can do it (small box for witness is sufficient), or Hyper-V, or StarWind ...

Many options - but you will need to get some more data on actual requirements & available options to spec out good boxes - for example a Z2 pool will not perform well for VMs, but would work fine as SMB storage for seq. reads (large images)... Also will 20 clients access large images (the same?) at the same time? or just 2 or 3 at a time? ...

 

[wildchild]

For Vsan you will at least need 3 hosts (Robo+Witness) unless you go virtual.

The question is - does it make sense to build a single point of failure? How soon do they really suffer if that one-box is gone? One week is unlikely but not impossible (think illness etc).

So If I were you I'd strongly consider to build something more reliable here, at least two hosts with appropriate redundancy. Vsan can do it (small box for witness is sufficient), or Hyper-V, or StarWind ...

Many options - but you will need to get some more data on actual requirements & available options to spec out good boxes - for example a Z2 pool will not perform well for VMs, but would work fine as SMB storage for seq. reads (large images)... Also will 20 clients access large images (the same?) at the same time? or just 2 or 3 at a time? ...

Z2 will perform just fine when using an all ssd pool for vm's.
Otherwise i must agree.. Why build a nee SPOF ?
You could build a cluster of 2 vmware machines (or really any other hypervisor that supports live migration, for that matter), passthrough the disk or jbod (make sure they are sas though when using 2 machine) and build your san/nas within that cluster.

If performance of that san/nas is an issue, or you want additional expansion , build 2 heads (xeon-d ??) and attach a jbod to both.
Run freenas on both heads and use carp, or run omnios and run their cluster software on it (additonal licensing required).

Why omnios ?
It's possible to get paid support on it, while i do not believe that is possible with freenas

 

[Rand__]

@Z2 pool - will depend on the SSDs I assume
Whats Carp?
And the 'paid' version of Freenas would be TrueNas.

 

[wildchild]

@Z2 pool - will depend on the SSDs I assume
Whats Carp?
And the 'paid' version of Freenas would be TrueNas.

Give he's building a proper storage platform, i would assume he's using enterprise rated ssd's.
Given that zfs mirrors are used for iops on spinning rust and ssd's will outperform those anyway, and he'll Probably max out his hba before getting anywere near maxing out his ssds...

30.10. Common Address Redundancy Protocol (CARP)

I know, but omnios doesn't require you to buy their support when it's not needed.
It's much cheaper as it is licensed on a processor socket bases, and it doesnt require you to purchases their (overpriced) sm hw, so you can build your own

 

[Rand__]

Ok, so Carp would take care of the addressing, but atm there is no easy way to sync/cluster two FN instances that I am aware off...
So I don't think that's a real option.

The next question is future expansion I guess - if they are expecting further growth then virtual storage (vsan, ceph etc) might be a good option since its easy to expand...

 

[wildchild]

Combine carp and hast..

 

[wildchild]

blather.michaelwlucas.com/archives/241

 

[Rand__]

Hm yes it might work, but does not really sound reassuring

"In learning about HAST, I more than once got the error “Split-brain detected.” This meant that I failed to consistently maintain one host as primary and one as secondary. To fix this, run “hastctl create mirror” on the backup node. This deletes and recreates the secondary HAST node. The primary will then sync the data to the backup, from scratch. For a 40GB pool, this took about 25 minutes. Both hosts were running on VMWare, however, and writing to the same disk. Be sure to test this with real hardware in your environment for realistic numbers. The lesson here is: don’t mirror anything larger than you must, because in case of trouble you must drag the whole image across the network to the other machine."

And given that FN dropped this feature due to the split brain issue I am not sure I would want to recommend it for a productive system

 

[wildchild]

Split brains can happen on ALL clusters, from firewall to storage appliances..
Just a matter of properly configuring and TESTING it.
What do you think xisystems are using ?
That being said..
Use nexenta or omnios and their ha plugin ( they use one and them same) if you need more reassurance, or something like zrep.
Point being, there is nothing special about freenas.
It's freebsd with a webgui, just like truenas,nas4free and others i may have forgot.
Even the compellant systems are hast and carp running on freebsd and their propriaitairy filesystem

 

[Rand__]

In the end the OP needs to decide whether its fit for his use case, but it is indeed an option

 

[gea]

My first goal would be: Keep it easy with the same as second goal.
Any HA solution will add a huge complexity and/or costs

Allow an outage of 30min with a simple manual failover and you must not use rocket science but simple tech than anyone that can build a pc can understand. For you VMs ESXi is a very good idea with the smallest footprint and the fastest recovery after problems. Beside you mainly need storage, as fast and secury as possible. This is why you should use server class hardware (ecc, LSI HBA, SSD, 10G Ethernet, http://www.napp-it.org/doc/downloads/napp-it_build_examples.pdf ) and a modern CopyOnWrite filesystem with checksums. While Microsoft is on a good way with ReFS its yet far behind ZFS regarding options and performance. So prefer a ZFS filer.

You can virtualize a ZFS filer under ESXi. I came up with that idea 7 years ago (http://napp-it.org/doc/downloads/napp-in-one.pdf ). You can even use ZFS via iSCSI for Windows. But resharing there via SMB is against rule #1 and rule#2 (keep it simple). If you use Solarish, you have full support of Windows ntfs alike permissions with full integration of ZFS snaps as Windows Previous Versions.

I would care about having two filers, virtualized or not where the second is the backup/ failover system. Keep them in sync with ZFS replication - down to a minute delay. As a ZFS storage appliance you can use Solarish where ZFS comes from or BSD where it was added years ago.

My top system regarding performance and features is Oracle ZFS - the mother of ZFS. But its not free and not compatible with OpenZFS (all other ZFS options). But there is the free Solaris fork Illumos where you get most of current Solaris with NexentaStor, OmniOS, OpenIndiana or SmartOS as main distributions. My top recomndation is OmniOS that I use as main platform for my napp-it.

Nothing against BSD or ZoL, but Solarish has simply the best ZFS integration with the OS and services like iSCSI, NFS and SMB fully integrated and part of ZFS from the beginning as ZFS properties and maintained by the OS maintainers Oracle or Illumos (developped by Sun, now Oracle and included in the free fork Illumos)

The fastest filer would be a dedicated filer but virtualized with HBA pass-through and similar RAM, you will get nearly the same performance. Use an SSD only pool for VMs (use Enterprise class SSD with powerloss protection) and a disk pool for data for costs otherwise also SSD only. If you want redundancy on the VMs and the data, use two ESXi AiO systems where you replicate storage and VMs between so the second box is a failover option for storage and VMs. On a failure of the primary system, you can start the VMs on the second system with all data near up to date or you can move all disks over, import the pool then the NFS datastore with VMs in ESXi and use original data.

Not complicated, easy to understand even under stress very fault tolerant.
You should only define software first and then the wanted hardware as hardware need and support is different among ESXi, BSD, Solarish or Windows.

 

[maze]

I know its a different ballgame. But have you considered buying a prebuild NAS box? Like synology or qnap..

Newer models have 10gig, and HA/Mirroring/cluster functionality is pretty standard on synology boxes.

Then do an iscsi vlan and Mount on your Vmware box(es). Would give you room to grow in host size

 

[KnottyMan]

My 3x R710s with 3x EQL 6x00E SANs when I was doing corporate worked great. Accountants wouldn't splurge for 10G so I had 12x1GbE per host, 8 per EQL. Talk about spaghetti, I was there every day though so whatever. But the BLINKY LIGHTS! For this client, I'm an hour away and a ferry from the site, so yes, simple = good.

I found a R720 with good procs, mem, and a H710 with SSDs (not sure of the brand) and vmware can manage the perc. So boom, one box, fast storage, known environment, half the price. Add some Mellanox to it and it's a fast AIO. It's not nvme, but do I need it? That's the other scary unknown, if I spend 2x on the new Supermicro, would it be vastly superior to SSDs on a 'legacy' SAS controller? Factor in my 20 clients on 1GbE.

Current plan: R720, 16x SSD 'spindles' on the perc, one VD for ESXi VM datastore, one big VD passed to the Windows VM for storage and native NTFS. Good enough speed since SSDs a'plenty. VM speed is not the issue since they're just little guys. SMB speed is the order of the day.

And I may be overthinking this too I'm realizing. Get a Supermicro of less stature, but nvme. Do Windows 2016 on bare metal, gasp, what?! Then a smaller Rsomething with a perc as a AIO vmware host and be done. Any monkey can manage the win box if I'm not there; vmware, pretty much same.

I've tested napp-it on Illumos, Omni, OpenIndiana, etc. Freenas is brand new to me as well. That's where the concern comes in, my noobness to the filers. I need more lab time. At work it was a LUN presented by the Equallogics, home stack was a Linux NFS.

I like the virt filer idea, the timing wigs me out. ESXi boots off SD/USB, filer then autostarts, boots off presented local SSD storage (but SPOF there - unlikely). Wait for it, then once timing has been sorted out and filer makes datastore available (via loopback iscsi?), then ESXi has to auto start more VMs now that the datastore is available? Seems easy to trip up, but cool you can do it, just like nested hypervisors. Here it adds complexity though and adds filer with noob admin.

I pondered three little SM Xeon-d super servers and vsan and such, but then cost/licensing goes exponential.

I've looked at the prebuilt NAS boxes. The ReadyNAS on the main page sparked my interest. Ehh... MD doesn't do it for me either.

 

[nk215]

IMHO all flash array on a 1gb network is a waste of $$. FreeNAS with good cache drives can handle that more than fine.

Don't forget backup and UPS.

I also wouldn't run hypervisor off of USB. It's just not reliable enough when the job is on the line. Get a small ssd for that.

I would just get a set of Synology boxes and do HA. They are small, quiet and reliable. There is no need to complicate a file server.

 

[KnottyMan]

Backup and power protection is handled since day one.

Agreed, sata DOM is the way to go. Never had an issue with my 710s and SD though. Or even my 1950s with good USB even.

I have not been impressed with Synology, Drobo, QNAP, etc. I've already got a linux box that does samba, etc, and that's what those are basically with less choice in hardware.

Yeah, all nvme is probably overkill with a 1GbE, that's why I'm struggling to come to a hardware decision. I want to make sure though that I have IOPS to go around for my 20 CAD machines, and if anything, get it back to showing that it's AutoCAD single threadness opening/regen that's the speed hit.

Probably just a pair of R720s, one vmware, one win2k8R2, with sata SSD on a perc will be a nice bump and accomplish what I need.

 

[wildchild]

My 3x R710s with 3x EQL 6x00E SANs when I was doing corporate worked great. Accountants wouldn't splurge for 10G so I had 12x1GbE per host, 8 per EQL. Talk about spaghetti, I was there every day though so whatever. But the BLINKY LIGHTS! For this client, I'm an hour away and a ferry from the site, so yes, simple = good.

I found a R720 with good procs, mem, and a H710 with SSDs (not sure of the brand) and vmware can manage the perc. So boom, one box, fast storage, known environment, half the price. Add some Mellanox to it and it's a fast AIO. It's not nvme, but do I need it? That's the other scary unknown, if I spend 2x on the new Supermicro, would it be vastly superior to SSDs on a 'legacy' SAS controller? Factor in my 20 clients on 1GbE.

Current plan: R720, 16x SSD 'spindles' on the perc, one VD for ESXi VM datastore, one big VD passed to the Windows VM for storage and native NTFS. Good enough speed since SSDs a'plenty. VM speed is not the issue since they're just little guys. SMB speed is the order of the day.

And I may be overthinking this too I'm realizing. Get a Supermicro of less stature, but nvme. Do Windows 2016 on bare metal, gasp, what?! Then a smaller Rsomething with a perc as a AIO vmware host and be done. Any monkey can manage the win box if I'm not there; vmware, pretty much same.

I've tested napp-it on Illumos, Omni, OpenIndiana, etc. Freenas is brand new to me as well. That's where the concern comes in, my noobness to the filers. I need more lab time. At work it was a LUN presented by the Equallogics, home stack was a Linux NFS.

I like the virt filer idea, the timing wigs me out. ESXi boots off SD/USB, filer then autostarts, boots off presented local SSD storage (but SPOF there - unlikely). Wait for it, then once timing has been sorted out and filer makes datastore available (via loopback iscsi?), then ESXi has to auto start more VMs now that the datastore is available? Seems easy to trip up, but cool you can do it, just like nested hypervisors. Here it adds complexity though and adds filer with noob admin.

I pondered three little SM Xeon-d super servers and vsan and such, but then cost/licensing goes exponential.

I've looked at the prebuilt NAS boxes. The ReadyNAS on the main page sparked my interest. Ehh... MD doesn't do it for me either.

So i'm guessing you're never going to do patching of that machine without planning for downtime ?
You're also gonna pay for a 7/24 support contract on that box ?
You're not going to patch your AOI without planning for down time ?

I think gea's idea of an active/passive repl setup isnt a bad one, just bear in mind, if this is a 24/7 env, double everything not only gives you peace of mind and an occasional worryfree holiday, it also allows for system management freedom.

Regarding the 10g, i'd pickup some intel 520's, they are well supported, can be had for scraps on ebay.
Couple that with dac's and just interconnect your hosts directly.
If the env grows, you can always buy 10g switches.

Also, be aware that passing through a jbod to more than 2 hosts might prove difficult without a sas switch.

 

[Net-Runner]

Hi,
I’ve been trying to build almost a similar setup for one of my customers’ a while ago and would like to share my experience. I hope it might be useful.
I would recommend you to avoid FreeNAS and similar stuff for production usage unless you’re are totally familiar with this product since it’s your primary storage and in this case, it is self-supported only. ESXi is a great choice, moreover, I would definitely recommend you looking towards two similar hosts and building a highly available cluster.
For a shared storage you do not need a SAN or NAS. It’s a single point of failure and directly attached drives are much faster. The most obvious option is VMware VSAN vSAN Software-Defined Shared Storage but it’s quite anemic in two-node setup and requires the damn witness. We are using Starwinds StarWind Software – StarWind Virtual SAN® – Starwindsoftware.com for this purposes because it is less expensive, works on top of hardware RAID which is very good because of improved performance and reliability (having a complete and consistent set of data in each host is priceless) and it does RDMA (unlike VSAN) which is very good too because we have Mellanox ConnectX3 and they work great with Starwind.
I’ve built such a setup by myself (SuperMicro-based) but requested the quotation for Starwind ready-nodes StarWind HyperConverged Appliance too. Surprisingly their price tag for similar DELL-based setup was not much higher.