VSAN storage help

BSDguy

Member
Sep 22, 2014
168
7
18
52
Hi All

I want to setup VSAN at home to teach myself more about the product and for my studies. Currently I have the following (single) server setup:

Supermicro X10SL7-F
32GB RAM
1 x Samsung Pro 850 256GB PCIe NVMe SSD (cache)
2 x Samsung Pro 840 128GB SATA (capacity)
1 x Samsung Pro 830 256GB SATA (capacity)
1 x Samsung Pro 850 512GB SATA (capacity)
ESXi 6 update 2

I enabled VSAN 6.2 on this server and changed the VSAN storage policy to allow only a single node in the "cluster" and the first thing I noticed was that that performance was terrible (even changing the stripe set to 4 didn't help). I'm using the onboard LSI-2308 controller for all the above drives (except the PCIe one) and it has a queue depth of 600. I'm not using any kind of RAID controller/card and would like to avoid this if I can.

After much reading it looks like my issue is down to running consumer SATA SSD drives. I was hoping a VMware guru could have a glance over the drives I would like to purchase before clicking the buy button!

I was thinking of the following options:

1) Single Samsung SM863 240GB Enterprise Class SATA SSD for the cache tier and then two Samsung SM863 480GB Enterprise Class SATA SSD drives for the capacity tier

If this is a good option and I get 200-400MB/s speeds then I am happy. Otherwise:

2) Intel DC P3700 Add-in Card PCIe 400GB for the cache tier and two Samsung SM863 480GB Enterprise Class SATA SSD drives for the capacity tier

These drives are on the VSAN HCL but I wanted to check that I am on the right track for setting up a good performance VSAN datastore. On my current setup I was averaging 20MB/s which is sad.

Further down the line (later this year) I'll be purchasing a Supermicro Mini Tower Intel Xeon D-1541 with 128GB RAM and will be reusing the above drives (in option 1 or 2) so could I connect the SATA SSD drives to the onboard SATA ports and use the Intel PCIe SSD for cache and get good performance? I am worried that the onboard SATA ports/disk queue lenght may be an issue. If it is then I need to rather look into using an M2/U2 drive for the cache tier so I can use a PCIe IBM M1015 HBA card for the SATA SSD drives (this card is on the HCL and has a good disk queu depth). The Supermicro Mini Tower Intel Xeon D-1541 only has a single PCIe slot. If this works then I'll purchase a further 2 nodes to make a 3 node cluster. This first node is the "POC" to make sure I get all the right kit.

Anyways, sorry for long post. Will be spending loads on this setup in the coming weeks/months (and its my first VSAN setup) so want to make sure I have the correct hardware for the storage.

Thanks!
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,324
1,785
113
CA
The problem I have with Samsung SSD is that their latency is 2x or more of Intel Enterprise SATA SSD. If you want snappy response, high IO, high sequential at this time Intel is the best choice. (Or for top end the Hitachi SAS 400GB w/intel inside but these are hard2find deals)

Currently you can't beat the Intel 400GB S3700 SATA for $160 on ebay (used). Depending how much space you need possibly 2-4x of those for capacity and a P3600 400GB NVME ($400-500) for cache would be a rocking setup for right around $1000.

NOTE: I'm not a vmware/vsan guru -- but in my experience with SSD that's route I'd go. (I have a p3600 400gb I'm tossing around the idea of selling or keeping too, if you go that route let me know and Ill see if I am selling or keeping -- if not there are others on ebay)
 

BSDguy

Member
Sep 22, 2014
168
7
18
52
Appreciate the comment!

I couldn't find any Intel 400GB S3700 SATA drives for sale on the UK site I use but I did find the "2.5 Inch 400GB SATA 3 Intel Data Center S3710 Enterprise SSD" but it costs £387 here. The "400GB HHHL (CEM2.0) Intel DC P3600 Add-in Card PCIe" is about £295. These are all new prices (ie: not second hand). Would these (similar) disks be ok still for VSAN? What about connecting the SATA drives to onboard SATA ports in the Supermicro Mini Tower server (worried about disk queue lenght).

Space wise I will need about 500GB but this is assuming VSAN dedupe/compression gets a 3:1 ratio. No problem as I can always add disks later on.

Is Intel my only safe choice here for good performance? Is it not even worth considering Samsung Enterprise drives? I've never owned either.

Thanks again!
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,466
621
113
Just out of curiosity what are your intentions with just the single node? It's my understanding you need 2 storage contributing nodes along with a third witness node at a minimum to really do anything with regard to vSAN.
 

BSDguy

Member
Sep 22, 2014
168
7
18
52
Just out of curiosity what are your intentions with just the single node? I'm pretty sure you need a 2 storage contributing nodes along with a third witness node at a minimum to really do anything with regard to vSAN.
Correct. You should run a minimum of 3 nodes but just to test my storage I set the VSAN storage policy to allow a single node so I could test. This is absolutely unsupported and shouldn't be done unless its for testing and/or in a lab.

I'm glad I did do this as it very quickly proved how bad my consumer SATA SSD storage is! When setting this up correctly (when I have bought the correct hardware) I would obviously have the 3 nodes setup in the VSAN cluster.

Edit: I saw your home setup: WOW! Is the disk performance awesome? Great setup you have.
 

BSDguy

Member
Sep 22, 2014
168
7
18
52
After having a look at prices and the VSAN HCL I am leaning towards the following Intel drives:

Cache tier: Intel SSD DC P3700 Series SSDPEDMD400G4 (400 GB, HHHL AIC)

Capacity tier: Intel SSD DC S3710 Series SSDSC2BA400G4 (400 GB, 2.5-inch)

Still wondering if I should consider any other makes? Considering the high cost of these drives I would probably start with one drive in the capacity tier to get started and to test before committing to buying more!
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,324
1,785
113
CA
The SM863 is a great improvement over the older generation Samsung Enterprise. For a home setup realistically if you don't need 'top of line' performance they would likely work just fine, and still provide great performance compared to consumer ssd ;)
 

BSDguy

Member
Sep 22, 2014
168
7
18
52
The SM863 is a great improvement over the older generation Samsung Enterprise. For a home setup realistically if you don't need 'top of line' performance they would likely work just fine, and still provide great performance compared to consumer ssd ;)
I don't need Ferrari speeds but I do need the storage to be snappy and I need things to happen fairly quickly. With my current disks it can take an hour or more to migrate a 40GB from one SSD datastore to another SSD datastore! I don't want to have to wait 2hrs to create a VM from a template or for a migration of a VM from one datastore to another. Been there done that...it sucks!

I would say if I can get 200-300MB/s speeds I would be happy. Anything over 300MB/s would be heaven compared to my pathetic 20MB/s currently.

Also, there won't be much load on this setup. Even though I plan on running up to 40VMs further down the line across the cluster, they won't be busy 99% of the time as I will be the only one using this setup for my studies/testing/etc.

The Intel drives look amazing but they sure are pricey. If I went the Samsung SM863 route would this work for me performance wise? Will there be any issues connecting these Enterprise SATA SSD drives to the onboard SATA ports on the motherboard?
 

BSDguy

Member
Sep 22, 2014
168
7
18
52
I couldn't help myself anymore so I ordered the Samsung SM863 480GB Enterprise SATA SSD this morning. It'll be interesting to see how the drive compares to my consumer drives when I do some testing late this week.

I'm going to try and use my consumer Samsung 850 Pro 256GB NVMe PCIe SSD drive as the cache tier and the Samsung SM863 in the capacity tier. Should be intertesting!
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,466
621
113
Correct. You should run a minimum of 3 nodes but just to test my storage I set the VSAN storage policy to allow a single node so I could test. This is absolutely unsupported and shouldn't be done unless its for testing and/or in a lab.

I'm glad I did do this as it very quickly proved how bad my consumer SATA SSD storage is! When setting this up correctly (when I have bought the correct hardware) I would obviously have the 3 nodes setup in the VSAN cluster.

Edit: I saw your home setup: WOW! Is the disk performance awesome? Great setup you have.
To be honest I haven't done much testing of the read write speed of the vSAN datastore other than the speed tests I've done between the mergerFS pool of NFS mounts I've created and some VMs.

What tools are you using to test your speeds? I'll run some and give you some of my own results.
 

aero

Active Member
Apr 27, 2016
328
70
28
52
I would like to know how you are testing as well. 20MB/s (or 11MB/s based on your estimate of 1hr for 40GB) does not seem right unless your disks are already very busy doing other things.
 

BSDguy

Member
Sep 22, 2014
168
7
18
52
To be honest I haven't done much testing of the read write speed of the vSAN datastore other than the speed tests I've done between the mergerFS pool of NFS mounts I've created and some VMs.

What tools are you using to test your speeds? I'll run some and give you some of my own results.
I was using ATTO Disk Benchmark and was also copying a 5GB ISO between VMs. The ATTO test results were fine but the real world performance was terrible.
 

BSDguy

Member
Sep 22, 2014
168
7
18
52
I would like to know how you are testing as well. 20MB/s (or 11MB/s based on your estimate of 1hr for 40GB) does not seem right unless your disks are already very busy doing other things.
My home lab is very quiet. I have (usually) 10 VMs running and most of the times things are idle. Only exception is when backups run, Windows Updates are being installed or I am installing a new product.

Interestingly, I updated the Avago/LSI 2308 VMware driver this weekend from version 19 to 20 and now my speeds are around the 80MB/s mark for all my SATA SSD datastores! Still not great but much better.

I'm hoping to be able to improve that even further with VSAN and Enterprise SATA SSDs. My goal are speeds in excess of 200MB/s...
 

aero

Active Member
Apr 27, 2016
328
70
28
52
Have you run baseline benchmarks on your storage with all VMs turned off? While they may be idle, they're still certainly performing some IO (logging mostly).
 

BSDguy

Member
Sep 22, 2014
168
7
18
52
How can I test disk performance if all the VMs are off?

FYI: Although I have 10 VMs running at any point in time they are spread out acorss 5 SSD drives/datastores.
 

aero

Active Member
Apr 27, 2016
328
70
28
52
In Linux with KVM or XEN hypervisors it would be quite easy to run iozone or bonnie on the datastores. ESXi is linux-like, but it doesn't look like you can run any of these linux benchmark apps directly on it. I'm not familiar enough with vmware to say if there is any way to test from the baremetal.

Perhaps a fair test would be to use a single VM to run the tests while all others are off.

How are your SSD's allocated? Raid?
 

BSDguy

Member
Sep 22, 2014
168
7
18
52
When I did the ISO copy test, I had two VMs running. One VM (and only one VM) was on the first Samsung Pro 840 128GB SATA datastore and the second VM was running on the other Samsung Pro 840 128GB SATA datastore (again, this was the only VM on this datastore). Both drives were connected to the SAS LSI-2308 ports on the motherboard. I then copied a 5GB ISO between the VMs and watched/timed the copy.

There is no RAID being used anywhere on this system so each drive is its own datastore.
 

aero

Active Member
Apr 27, 2016
328
70
28
52
Ah, I understand your test scenario now. The 80MB/s ATTO on each datastore seems pretty terrible. Are CPU cores pegged while it's running? Low on memory?

For the transfer between VMs, what vNIC type are you using? vmxnet should be the best performing. I would expect that you should be able to hit your max disk speeds on such a transfer, but it doesn't look like you get anywhere close to that.
 

BSDguy

Member
Sep 22, 2014
168
7
18
52
When running an ATTO benchmark the results are really good 300+MB/s for write speeds and 500+MB/s for read speeds.

The 80MB/s speed I get is when doing an actual file copy between VMs (with the 5GB ISO).

All my VMs are using the VMXNET3 vNIC and in Windows the NIC speed is showing as 10gbps.
 

BSDguy

Member
Sep 22, 2014
168
7
18
52
I've just migrated a 24GB VM from my PCIe NVMe SSD datastore to the one SATA SSD datastore and it took about 9min for the migration to complete. About 43MB/s. Sad!