Need a SAN upgrade for our FreeNAS & Xen-XCP Cluster - Starwinds VSAN?

Indecided

Active Member
Sep 5, 2015
128
53
28
We're trying to decide if we should move our FreeNAS machines to a HA Starwinds VSAN.

This is a bit of uncharted territory for us as we've historically been a 100% Linux/BSD shop all the way and as our direction now is to have HA for our main cluster, we've been struggling to find a Linux equivalent (software only).

While Starwinds has a converged model on Linux/ESXi, it doesn't work for us as our VM nodes are all built on Xen/XCP, plus we're not really looking to go HC right now.

Eventual setup would be a 16TB, all-flash, HA cluster for storage pushed over 40GBe to a cluster of about 16 VM nodes over 10GBe each.

So this is what we're considering now. We had a trial setup but haven't had much time (ironically given the lockdown) to really tinker with it, but i'm wondering if i'm missing out on any other alternatives.

We're only considering software options at this time - so that puts TrueNAS and other hardware-first solutions out of the ballpark. The significant factors are cost at the end of the day, and we have enough hardware sitting around to pretty much cobble up whatever we really need.

VSAN will run ~10k for our 2 nodes. That's pretty much all I can get approval for budgets right now given that we've been running with 0 software costs for the past 8 years with Xen Standalone - then FreeNAS. But with that all considered, I think a HA SAN solution is what we really need going forward, but i'm just not sure - or perhaps not 100% comfortable with a Wintel platform for a SAN. Probably just need some convincing?
 

Net-Runner

Member
Feb 25, 2016
80
24
8
37
As an MSP, we have a bunch of customers using Starwind products as hyperconverged clusters and as pure storage appliances. Most of them are Hyper-V/Windows-based and work pretty great. The hyperconverged configuration seems a bit more complicated and might have some Microsoft-related caveats. Dedicated storage, if appropriately configured, works like a machinegun. If I remember it correctly, you can use their Linux version instead of Windows-based if you don't like the Wintel platform and want to avoid additional licensing costs. Install the free ESXi on your storage servers and run their Linux version on top of them. We have a customer that currently runs precisely the same scenario for almost half a year already. As far as I know, it works and performs excellently.
 
  • Like
Reactions: Indecided

Indecided

Active Member
Sep 5, 2015
128
53
28
As an MSP, we have a bunch of customers using Starwind products as hyperconverged clusters and as pure storage appliances. Most of them are Hyper-V/Windows-based and work pretty great. The hyperconverged configuration seems a bit more complicated and might have some Microsoft-related caveats. Dedicated storage, if appropriately configured, works like a machinegun. If I remember it correctly, you can use their Linux version instead of Windows-based if you don't like the Wintel platform and want to avoid additional licensing costs. Install the free ESXi on your storage servers and run their Linux version on top of them. We have a customer that currently runs precisely the same scenario for almost half a year already. As far as I know, it works and performs excellently.
I was told by Starwind support that the ESXi version isn't applicable for "storage separate" solutions. Meaning to say, they only support it if you're using it in the HCI model where your VMs are on ESXi and passed-through ala Nutanix, but not as a HA-active-active type iSCSI SAN where compute is separate.

Having said that, i'm probably more favorable about having the SAN software sit directly on the host versus being virtualized-and then passed through, IMHO it feels like another layer of complexity to diagnose when something goes wrong.

I guess i'm just trying to validate that over the long run Starwinds will indeed be the right choice for our SAN "cluster" if you can call it that, or consider any other solid software platforms if there are any. I don't think we can go back to FreeNAS or any non "hot" HA capable software platforms because we can't rely on restoring backups for business continuity.
 

Net-Runner

Member
Feb 25, 2016
80
24
8
37
At first glance, a Windows-based bare-metal installation where virtual SAN software sits directly on the physical level might seem more straightforward. It probably is. If that is a reasonable justification for you to purchase two Windows Server licenses, let it be so. I can only reassure you I have seen multiple configurations of that kind in the wild, and they work great, no worries here.

I personally love a virtualized approach much more. First of all, it gives you an excellent storage abstraction level, which means if one of the storage controllers fails for some reason, it is much simpler and quicker to restart just the storage virtual machine instead of rebooting the whole server. This takes maybe 30 seconds and can be automated in ESXi using a heartbeat sensor, which makes the entire solution even more redundant. ESXi (or KVM) used as a foundation for this approach is much more credible to me than Windows Server. Just think about regular updates and patches that require a server reboot. This is not an issue with Starwinds since it keeps working while one of the controllers is rebooted, but the procedure still requires numerous efforts almost every week.

I know a large ISP here in Germany with whom we cooperate that has this scheme on top of free ESXi currently running and feeding storage to 10+ other hosts with various OSes/hypervisors. You should probably ask starwinds support regarding this case once again.
 

Rand__

Well-Known Member
Mar 6, 2014
4,578
911
113
I assume you are using none of the ZFS inherent goodies like checksumming, snapshots or replication? And also I assume you dont mind changing to hardware Raid?

I mean its all fine, but FreeNas and Starwind are actually quite different beasts in my opinion so its surprising to be so casual about switching over :)
 

Indecided

Active Member
Sep 5, 2015
128
53
28
@Net-Runner : I would have preferred this simply due to familiarity despite the VM passthrough, but they said for my use-case it wasn't possible. I'll speak to them on this.

@Rand__ : Yes, we have used snapshots and tried replication before, but frankly we haven't seen the need to restore _yet_ it over the last several years, which speaks tons for FreeNAS's reliability. However, we recently had several failures not in the SAN layer but at the VM nodes which for some reason did _not_ trigger the Xen HA mechanism and resulted in us needing to manually detach, destroy and rebuild the node itself.

This got us thinking about other parts of the system - that while we could have a cold spare or even a warm FreeNAS spare sitting around using replication, it couldn't give us the expectation that a warm to hot failover could be completed in 5 minutes. Having said that, we would also have to restart the VMs. That's pretty much a hard NO now for us - 5 years ago it was quite different, with less money and less reliability expected or needed.

Exploring Starwinds, we are able to setup iSCSI multipathing to two or more nodes. We've not done super in-depth testing of our POC cluster yet, but our understanding and expectation is that in the event one node goes down, everything should continue to work as Xen/etc will simply pick up the second SAN node via the other iSCSI path.

Basically - we need to build out resilience at every layer - not just spares, but continuity without needing to hop in car at 5am in the morning and drive 45 minutes to our colo to swap something out, or run some takeover scripts. It should simply keep going - with a short pause. Ala HA.

As for hardware raid, i've never been a big fan since we've always been software based RAID people - early on our physical nodes were MD raid based, and the we moved on to FN with ZFS. I couldn't say if there are any big pros to using HW raid (besides being required to do so) and losing all the fancy features of FN - but with our current business alignment, reliability - not speed and features - are considered paramount above all.

So - that's why we're considering a HA SAN solution.
 

tjk

Active Member
Mar 3, 2013
285
48
28
www.servercentral.com
Take a look at TrueNAS appliances, they have some killer pricing specials right now, and you get HA.

I love Starwinds, however I just did a POC using SW, iSER, and VMW 6.7 cluster...in my testing it worked well for about 5 days while I did IO stress testing. I stopped the testing and just let things sit for 2 days, came back and SW had crashed on each node, wasn't able to restart the process, reboots didn't help. Ended up losing the test VM's.

I need to re-create this and get support involved, just haven't had time to play around with this again.
 

Indecided

Active Member
Sep 5, 2015
128
53
28
The main reason why we've looked away from traditional storage & even TrueNAS is as mentioned - we have an essentially ever-ongoing surplus of hardware in our testing labs. So we probably have 3 or 4 of the same machine sitting around.

We would need to go with the X20 as we've moved off spinning rust for VM nodes alltogether. @ 700/TB... I probably will get shouted at as we have several hundred TB of SAS SSDs that we go through every week at a fraction of that price. So it wouldn't quite make sense to purchase new storage or hardware.

Also, our core DC isn't in the US, so that doesn't help with AHR/RMA at all unless the hardware happens to be Dell/HP OEM'd/whitelabel'ed with manufacturer support.

Henceforth a software-only solution is what we're looking at. I guess we really have to give it a full-on POC test-run as right now we're more or less doing hardware burn-in testing while the nodes are idle. I guess we'll spin up some complex workloads and keep on failing the cluster nodes over.

@Rand__ - nope - we will give it a look. That's what we've been looking for, options :)
 

tjk

Active Member
Mar 3, 2013
285
48
28
www.servercentral.com
The main reason why we've looked away from traditional storage & even TrueNAS is as mentioned - we have an essentially ever-ongoing surplus of hardware in our testing labs. So we probably have 3 or 4 of the same machine sitting around.

We would need to go with the X20 as we've moved off spinning rust for VM nodes alltogether. @ 700/TB... I probably will get shouted at as we have several hundred TB of SAS SSDs that we go through every week at a fraction of that price. So it wouldn't quite make sense to purchase new storage or hardware.

Also, our core DC isn't in the US, so that doesn't help with AHR/RMA at all unless the hardware happens to be Dell/HP OEM'd/whitelabel'ed with manufacturer support.

Henceforth a software-only solution is what we're looking at. I guess we really have to give it a full-on POC test-run as right now we're more or less doing hardware burn-in testing while the nodes are idle. I guess we'll spin up some complex workloads and keep on failing the cluster nodes over.

@Rand__ - nope - we will give it a look. That's what we've been looking for, options :)
Let me know if you find anything, I've been on the search for a good bare metal software san for years, was hoping Starwinds was it, but the failure during the POC wasn't comforting.

StorPool is another you way want to look at, and they want you to use flash/SSD for everything. ZetaVault was another decent solution I tested a year ago, but it appears they are dead or at least appear dead now. Open-E also has a ZFS based SW solution, and you can do HA with it. I used the DSS SW as a backup target many years ago (nfs, cheap and deep storage) and it was reliable, this was before they switched to a ZFS core offering.
 

Rand__

Well-Known Member
Mar 6, 2014
4,578
911
113
I used the DSS SW as a backup target many years ago (nfs, cheap and deep storage) and it was reliable, this was before they switched to a ZFS core offering.
Is that good?
Got a license for for that with a pair of servers I bought a while ago but never ran it since I was looking at a ZFS solution instead
 
  • Like
Reactions: tjk

tjk

Active Member
Mar 3, 2013
285
48
28
www.servercentral.com
Is that good?
Got a license for for that with a pair of servers I bought a while ago but never ran it since I was looking at a ZFS solution instead
The DSS (non ZFS) product is good for slow storage, no caching, etc. We used it as a multi PB repo for VM backups for years w/o problems.

Have not spent a lot of time with the ZFS (Jovian) product yet, they have a bunch of different options for HA with Jovian, Eth sync, shared SAS, FC, etc.
 
Last edited:
  • Like
Reactions: Rand__

Rand__

Well-Known Member
Mar 6, 2014
4,578
911
113
So not particular fast? Thought it might be okish since they use HW Raid iirc
 

LaMerk

Member
Jun 13, 2017
34
4
8
29
@Net-Runner : I would have preferred this simply due to familiarity despite the VM passthrough, but they said for my use-case it wasn't possible. I'll speak to them on this.

@Rand__ : Yes, we have used snapshots and tried replication before, but frankly we haven't seen the need to restore _yet_ it over the last several years, which speaks tons for FreeNAS's reliability. However, we recently had several failures not in the SAN layer but at the VM nodes which for some reason did _not_ trigger the Xen HA mechanism and resulted in us needing to manually detach, destroy and rebuild the node itself.

This got us thinking about other parts of the system - that while we could have a cold spare or even a warm FreeNAS spare sitting around using replication, it couldn't give us the expectation that a warm to hot failover could be completed in 5 minutes. Having said that, we would also have to restart the VMs. That's pretty much a hard NO now for us - 5 years ago it was quite different, with less money and less reliability expected or needed.

Exploring Starwinds, we are able to setup iSCSI multipathing to two or more nodes. We've not done super in-depth testing of our POC cluster yet, but our understanding and expectation is that in the event one node goes down, everything should continue to work as Xen/etc will simply pick up the second SAN node via the other iSCSI path.

Basically - we need to build out resilience at every layer - not just spares, but continuity without needing to hop in car at 5am in the morning and drive 45 minutes to our colo to swap something out, or run some takeover scripts. It should simply keep going - with a short pause. Ala HA.

As for hardware raid, i've never been a big fan since we've always been software based RAID people - early on our physical nodes were MD raid based, and the we moved on to FN with ZFS. I couldn't say if there are any big pros to using HW raid (besides being required to do so) and losing all the fancy features of FN - but with our current business alignment, reliability - not speed and features - are considered paramount above all.

So - that's why we're considering a HA SAN solution.
That's basically how Starwind works. Since their Linux-based version does not support the "storage-separate" converged scenario, I have made V2P and run installed it bare-metal on two separate servers. From what I faced, the storage subsystem performs the same as being a passthrough to Starwind virtual appliance.

I see they added ZFS support to Starwind and allow setting up HA on top of it. I haven't tried it yet, however, planned on the quarter. Check if it would work for you.