Looking for a good SAN solution for VMware

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

XeonSam

Active Member
Aug 23, 2018
159
77
28
On a small budget, have looked into different possibilities.

Vsan - too expensive and storage usage is too small (usable storage). Performance is great tho.

OSNEXUS - This was going to be the solution as it is free and has all the features of ZFS and supposedly Rdma support (found out this has been dropped). Makes it just as attractive as freenas.

Starwind - Not too excited about this solution as it's all windows based. The Windows dependency complicates everything with licenses and CALs... Also, only iscsi support is available (no iser/rdma support)

Windows 2019 Storage spaces - Looking to isolate the storage from the VMs for easier management. Also.... It's windows, not really confident in using it as a SAN solution. Any good feedback on small scale roll outs?

Openfiler - Can't seem to find anything positive about this one.

Thanks for any feedback in advance
 
  • Like
Reactions: Net-Runner

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
Its probably going to help if you let us know what your requirements are... as in
-performance (#of VMs, type)
-service (pre made appliance, roll your own, service contract, self service...)
-HA requirement or single box ...
 

NISMO1968

[ ... ]
Oct 19, 2013
87
13
8
San Antonio, TX
www.vmware.com
VMware vSAN can do RAID5/6 since... forever? ...so I don't see why you're complaining about low usable capacity, it's definitely on the level. Performance is on the slow side compared to the other other guys though. Good thing is it's in kernel and supported by hypervisor vendor so one throat to choke when it comes to support.

OSNEXUS is mom and pop shop, CEO is doing support and replies to quote requests. Any Linux ZFS distro you like will do the trick.

StarWind. They do have Linux version in terms of VSA and they do iSER since 2010 maybe so I'd suggest you redo your home work and train GoogleFu (VMware vSAN isn't different).

Storage Spaces aren't supported inside the VMs (only in Azure) so you're on the "unsupported" death row again. S2D doesn't shine but it has the same benefits like VMware vSAN, if you have Premier support with Microsoft you'll find somebody to return you support calls :)

OpenFiler is dead in the water.

On a small budget, have looked into different possibilities.

Vsan - too expensive and storage usage is too small (usable storage). Performance is great tho.

OSNEXUS - This was going to be the solution as it is free and has all the features of ZFS and supposedly Rdma support (found out this has been dropped). Makes it just as attractive as freenas.

Starwind - Not too excited about this solution as it's all windows based. The Windows dependency complicates everything with licenses and CALs... Also, only iscsi support is available (no iser/rdma support)

Windows 2019 Storage spaces - Looking to isolate the storage from the VMs for easier management. Also.... It's windows, not really confident in using it as a SAN solution. Any good feedback on small scale roll outs?

Openfiler - Can't seem to find anything positive about this one.

Thanks for any feedback in advance
 
  • Like
Reactions: XeonSam

Net-Runner

Member
Feb 25, 2016
81
22
8
40
Vsan - too expensive and storage usage is too small (usable storage). Performance is great tho.
Second this one. It is damn expensive, storage efficiency is questionable, performance is adequate, but only up to a certain level (is capped). The vSphere integration is impressive, though. And it is stable as hell if configured and managed properly.

OSNEXUS - This was going to be the solution as it is free and has all the features of ZFS and supposedly Rdma support (found out this has been dropped). Makes it just as attractive as freenas.
Unlike FreeNAS, I didn't have a chance to see OSNexus in production. FreeNAS is an excellent solution if you are OK with using entirely self-supported storage in production.

Starwind - Not too excited about this solution as it's all windows based. The Windows dependency complicates everything with licenses and CALs... Also, only iscsi support is available (no iser/rdma support)
We have multiple customers that are using both Windows-based and Linux-based versions of Starwinds, and from what I know, it works great. It also fully supports iSER/RDMA StarWind VSAN & VTL features in a nutshell. I know for sure since I have been helping one of our customers to configure his new Mellanox ConnectX5 cards recently to be used with Starwinds. They are currently working on NVMeoF implementation; thus, I think your impression is a bit outdated.

Windows 2019 Storage spaces - Looking to isolate the storage from the VMs for easier management. Also.... It's windows, not really confident in using it as a SAN solution. Any good feedback on small scale roll outs?
I would avoid using Storage Spaces unless you are purchasing some kind of ready nodes from Microsoft Cert/Gold/whatever-they-call-it partners. Too many issues reported regarding Storage Spaces, ReFS, Storage Spaces Direct, etc.

Openfiler - Can't seem to find anything positive about this one.
No idea on this one, but the website and screenshots look pretty outdated.
 
  • Like
Reactions: NISMO1968

XeonSam

Active Member
Aug 23, 2018
159
77
28
Thanks for all the feedback!

I tried vSAN with all SSD's backed with NVMe with the cache layer. Performance was amazing considering it was a 10G setup.
The IOPS were basically local storage speeds and out performed a regular setup with iSCSI and FC using ZFS.

iSER (RoCEv2) is supported in ESXI which was why I was siding towards a linux distro like OSNEXUS. I contacted their APAC contact and they say they no longer support RDMA as the need didn't justify the investment.


Starwind seems promising, but with the hardware budget (all flash) I doubt there will be much left over for extended support. If there was a pay per support ticket or had something perpetual like Windows Server I think it would get a go.

--> Has anyone had success in setting up iSER on the initiator side (Windows Storage Spaces) and the target (ESXI 7 or 6.7) using Mellanox Connectx-3 hardware?

1. I've successfully installed the drivers on both ends but am still just getting 10GbE rather than 40GbE with RDMA from Windows.
2. Using direct connect and old Connectx-3 NICs I'm realizing there is less support on Mellanox homepage as most of it refers to Connectx4/5.
- I just started my GoogleTrek on infiniband and maybe lacking some basic knowledge that everyone should know
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
So you have any numbers and measurement criteria for us?
My personal experiences with vsan have not been that good, so I'd be interested in both HW and the above test settings to see whether your's are in line :)

Not been having success with RDMA, but I can tell you no need to google IB if you are using ESXi, its not supported any more.
 
  • Like
Reactions: NISMO1968

XeonSam

Active Member
Aug 23, 2018
159
77
28
You got to be kidding! RoCEv2 is still supported. The drivers exist for 6.7 and 7.0 and documentation for it exists. Do you have somewhere this can be verified?
 

XeonSam

Active Member
Aug 23, 2018
159
77
28
I thought Infiniband was the technology while iSER a protocol with 2 different versions: RoCE and iWarp. Both using RDMA as the transport method.
This is confusing when used to iSCSI and FC... things were so black and white then.


So you're saying RDMA isn't supported by VMware? A simple google search will show support for RDMA. Then if infiniband isn't supported, RDMA wouldn't be either.

I feel like there's a disconnect. Trying to wrap my head around this.
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
RDMA is supported in the form of RoCE/v2

As of now its unclear which parts of vSphere actually can use RDMA, but iSer (ie iSCSI/RDMA) seems to be the most likely as its explicitly mentioned.
iWarp probably works too as there is a Chelsio iWarp driver for vSpherem but also unlclear whether this will only be useful for iSer or also vSphere internal traffic (or other external connectivity like NFS/NVME over RDMA)
 
  • Like
Reactions: XeonSam

XeonSam

Active Member
Aug 23, 2018
159
77
28
So you have any numbers and measurement criteria for us?
My personal experiences with vsan have not been that good, so I'd be interested in both HW and the above test settings to see whether your's are in line :)

Not been having success with RDMA, but I can tell you no need to google IB if you are using ESXi, its not supported any more.
Sorry, I didn't do benchmarking tests, however, I used a test lab with 3 hosts. All limited to copper at 10G. vSAN was just so much more snappier than when using a SAN box. I was using ZFS on the SAN box and tested both iSCSI and NFS. iSCSI wasn't multipathed.

Second test was in an enterprise production environment with an EMC all flash backend; 16Gb FC. This was multipathed... and the latency was noticable compared to the vSAN 3 node test lab. I could be mistaken as we're taking apples to oranges, but my perception was that vSAN was running on local storage while the enterprise production was running on remote hardware. Sequential was amazing, but that was just about it.
 

XeonSam

Active Member
Aug 23, 2018
159
77
28
RDMA is supported in the form of RoCE/v2

As of now its unclear which parts of vSphere actually can use RDMA, but iSer (ie iSCSI/RDMA) seems to be the most likely as its explicitly mentioned.
iWarp probably works too as there is a Chelsio iWarp driver for vSpherem but also unlclear whether this will only be useful for iSer or also vSphere internal traffic (or other external connectivity like NFS/NVME over RDMA)
Thank God. Or else I would have really screwed up (long story)
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
Sorry, I didn't do benchmarking tests, however, I used a test lab with 3 hosts. All limited to copper at 10G. vSAN was just so much more snappier than when using a SAN box. I was using ZFS on the SAN box and tested both iSCSI and NFS. iSCSI wasn't multipathed.

Second test was in an enterprise production environment with an EMC all flash backend; 16Gb FC. This was multipathed... and the latency was noticable compared to the vSAN 3 node test lab. I could be mistaken as we're taking apples to oranges, but my perception was that vSAN was running on local storage while the enterprise production was running on remote hardware. Sequential was amazing, but that was just about it.
Well if you used zfs then it totally depends on the drives and or slog being used, that can go from a crawl to a run with "ZFS on Flash".
Also the next thing depends on the amount of users you are going to run - vSan will work best with lots of users, bad with few.

Also I am surprised that FC was not snappy, its main benefit is the low latency (as far as i know, never used it, just read about it)
 

XeonSam

Active Member
Aug 23, 2018
159
77
28
Well if you used zfs then it totally depends on the drives and or slog being used, that can go from a crawl to a run with "ZFS on Flash".
Also the next thing depends on the amount of users you are going to run - vSan will work best with lots of users, bad with few.

Also I am surprised that FC was not snappy, its main benefit is the low latency (as far as i know, never used it, just read about it)
Yea.

ZFS really is all about the number of drives and Zlog; and also settings. It was all flash on RAID0 (striped) using HGST SAS3 drives. Same drives which were used in vSAN. No Zlog was used but everything was tweaked for performance. ZFS is great, dedupe and compression takes very little overhead compared to vSAN's native compression settings.

FC16 is great, don't get me wrong. But vSAN with all flash was just noticeably better. Performance was basically local storage speeds. The cost for the number of drives was just too much of a limiting factor.
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
well let me say, my (untuned except HW) ZFS filer (no RDMA) has better performance with SAS3 SSDs & SLOG vs my all NVMe vSan (single disk group) for my low user count workload... but to each their own :)