Storage Performance advice

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
M

modder man

Active Member
Jan 19, 2015
657
84
28
32
Looking for some pointers from some of you who are much better with these things than myself. As of recently I have been seeing 30ms spikes of write latency on my VM datastore which is backed by a 960GB Samsung 853T. I would think this disk should provide enough IO for lab VM's though there are currently 34 of them on the host. The second problem is that this disk is about full so I am needing to get some more space. I also have a freenas VM on this host that holds 12 4TB disks in 3x 4x4TB raidz1. This are currently only used for bulk storage as I am sure I cannot get enough IO out of these to be good for anything else.

Was hoping some of you may have some thoughts or advice on how to get the best performance out of what I have.

I have a 100GB intel S3700 laying around plus the disks in the host. I cannot add disk enclosures or a larger chassis at this time and all 12 bays in the front are full. So if I am adding disks they will be sitting on top of the fan shroud in the chassis.
 
T_Minus

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,640
2,057
113
IMHO Samsung before the latest generation are all > latency than Intel & HGST drives, now load up 34 VMs and fill near capacity and of course it's going to struggle even more. Samsung PM853T SSD Review | StorageReview.com - Storage Reviews tests there and Samsung PM853T 960GB Enterprise SSD Review show up to ~15MS latency in short bursts, with yours 24/7 I'm sure it's getting worse/struggling more.

You really need to add more disks for 34 VMs than 1 full SSD.
It's not a cheap solution but 6x of those 'deal' S3710 400gb in RAID10 would be great or pool of mirrors in ZFS and then either get a 200GB S3700 SLOG or try without, I don't think your 100GB will keep up with an SSD pool.
 
M

modder man

Active Member
Jan 19, 2015
657
84
28
32
I didnt think the S3700 would work to well for an SSD pool. The 960GB 853T is currently serving as local storage. Part of me was wondering if putting the S3700 on the bulk pool would allow be to put some of the fatter VM's on that pool to ease some of the load on the main SSD. Trying to keep the costs down as much as possible. I suppose I ask the questions as much to see what I can learn as much as fixing the problem if that makes sense. I have come to a point that I have realized that I dont know how those of you with heavy IO loads set up the storage to run that.

Unfortunately that sale has ended and those guys were even local I could have picked a few up in person.
 
Last edited:
T_Minus

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,640
2,057
113
Lots of IO... More drives :D

I would try out what you said if you can easily... add the S3700 100GB as a SLOG to the 'storage' pool you have, and move a couple VMs there and see how they work. Maybe it's enough to free up the SSD so it keeps working how you need :)

Another option might be to snag one of the S3710 400GB for $155 and simply use it for local storage for 3-5 of your 'heavy usage' VMs... it will have around 2x the write IOPs as that Samsung with lower latency so it will be faster overall for writes, and faster for mixed workloads. The same seller may be gone, but there's at least 2 still with them 'dealing', and if you can't get them you can get some $120 400GB HGST SAS SSD which are essentially S3700 with a bit more endurance, and slightly > perf. iirc in one area or another, and, even a bit lower latency.

Realistically though, to handle writes you need disks which cost money, or expensive SLOG and cache drives which cost money.

An interesting solution... due to recent price decreases.... what if you got a $350 ZeusRAM SLOG device, and used your current 960GB SSD for Cache? This would allow you to use all of your 'storage' space you need, have a very fast write and very good read on your common files. With the 3x striped depending on drive could yield some good sequential too.
Just throwing out another idea :)

OR, another thought... can you fit PCIE drives? Could go PCIE Fusion IO or NVME too.
 
M

modder man

Active Member
Jan 19, 2015
657
84
28
32
use the 960GB as a cache in what way? as in Vmware host cache? the striped storage drives are just cheap 5900rpm desktop drives. I did not get those with any intention of ever putting VM's on the just exploring what options I have at this point. I did no thing I would run into much IO issues even though I only have once SSD since it is just a lab. I do have space where I could stick a PCIE drive if I found the right deal to jump on.
 
T_Minus

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,640
2,057
113
modder man said:
use the 960GB as a cache in what way? as in Vmware host cache? the striped storage drives are just cheap 5900rpm desktop drives. I did not get those with any intention of ever putting VM's on the just exploring what options I have at this point. I did no thing I would run into much IO issues even though I only have once SSD since it is just a lab. I do have space where I could stick a PCIE drive if I found the right deal to jump on.
Click to expand...
L2ARC for the 960 if you use the entire storage pool for VMs, which you could do with a very fast SLOG, and a nice amount (960gb) of cache. I was getting 4K random 350MB/s+ with ZeusRAM and WD RED in 1 single raidz2 pool with a bad configured network setup... I haven't gone back to re-test now that I've worked out some bugs but I'd imagine with your 3 striped vdevs you could pull that much or more with a ZeusRAM or NVME.

There's been some Intel P3600 400GB NVME on ebay -- something you may just throw in there and move your high-IO drives too. It won't win any performance contests but it's def. better than that 960gb.

Another option is to find a 160 or 350GB Fusion-IO SLC drive, and use that for a slog they can be found $200 or less, even the MLC version would likely be fine and even cheaper. And then use your 960gb as L2ARC and migrate your VMs to the 'storage' pool, that's now actually a very-capable VM storage.
 
M

modder man

Active Member
Jan 19, 2015
657
84
28
32
So it sounds like in your experience with a properly configured l2arc and a SLOG even a raidz array can provide plenty of performance for VM's? That is opposite of what most say. That said I have seen some of your build logs I trust the advice you give.
 
T

ttabbal

Active Member
Mar 10, 2016
747
207
43
47
From the context, it sounds like he's talking about using it as a ZFS cache device. Combined with a fast SLOG, even those 5900 drives can get decent numbers. A lot of the I/O would hit the SSDs, with the backing spinners updated in bursts. For this to work well you need to get good devices for the SLOG and cache. Pay attention to the models T_Minus calls out, it matters more than you might think.

If you prefer to stick to SSD only for the VM store, you would need to get more SSDs and stripe them. I think a big part of your problem is that it's filling up. SSDs don't perform well when full, latency goes up as the controller has a harder time locating blocks to use. I would get enough drives that you're completely replacing the old device, then copy the contents to the new array so that it can balance across the devices. That should help spread the load out over the most possible disks, which is the best way to get more performance. If you keep good backups and don't mind some downtime to restore, you could run them non-redundant. If not, use striped mirrors (RAID10).
 
T_Minus

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,640
2,057
113
This is a "throw it in" and test, not using vmxnet3, not using jumbo frames, no configuration tweaks... just a 'raw' test. I didn't re-test with the ZeusRAM but I did with the 12Gb/s SAS SSD and it made a HUGE difference so for discussion sake assume these numbers are still low on what you can get with 5900RPM drives.

ScreenClip.png

Keep in mind this is a SINGLE VDEV pool, yours is 3x striped too so the flush time to your drives will be faster than mine.

You really have 2 options that can be done a bunch of ways:
1- Setup your "storage" pool to handle VMS by adding a very fast SLOG (ZeusRAM would be ideal, FusionIO #2, and only NVME #3 because I don't think it's supported yet in pass-through on ESXi which is why I haven't used/tested it in my ZFS AIO home lab either.) and use your SSD as L2ARC cache drive

2- Add more SSD to your current SSD and run RAID10 if you need the mirrors or RAID0 if you can handle failure and rebuild from backup. As a note to this option, for a home setup, you can get Intel 2.5" or 3.5" drive cages with fans built-in for $150 or less and simply stick it on top your tower and run a 8087 x 2 to fanout out a back PCIE slot, or if the cage has an expander (not as ideal for performance) 2 SAS cables and be done with it.

2a - Since you have room for a PCIE device you could add a 9300-8E or 9200-8E (6gb/s or 12gb/s) HBA with 2 external ports, then connect those to your 'outside' intel cage, or, if you have the room and spare parts you cuold configure a jBOD :) SuperMicros JBOD3 makes the fans controllable, IPMI access, and more.


Right now for home I'm using 8x SSD for my VM storage, and I have some Intel P3700 800GB NVME for testing local storage only. I'm hoping eventually I can add NVME pass-through to my OmniOS/NAPP-IT VM and see how that works, as I too would really like to have more space for VMs in test/dev that don't need the 'high performance'... or rather, don't eat into my really expensive storage ;) LOL!! For reference, for my business 'general' VMs not actual "working vms" just general stuff to keep biz going I found it cheaper to go with 24x Intel 120GB S3500 (this was months ago) now I'd go with the 300GB for $80 you can score! $30 more than I paid for over 2x capacity and performance. You don't need to buy the BEST Intel SSD if you're going to run 4, 8, 12, etc... of them, the S3500 IOPs start to add up, and if you're not doing tons of writes the 'spread' of IO around to drives really helps with the 'clean up' routines, and preventing performance to tank for too long of a time.
 
  • Like
Reactions: Cole
M

modder man

Active Member
Jan 19, 2015
657
84
28
32
I dont necessarily prefer to stick to SSD only for VM storage that just seemed to make the most sense with the current hardware that I have. The ZeusRAM is probably a bit more than I would want to spend but I would/could consider grabbing one of the fusion io drives. The server is currently racked in a datacenter which cuts down on my ability to add a disk shelf or anything else external.
 
T_Minus

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,640
2,057
113
Ahhh... in that case what about future proofing...

Ggetting a SuperMicro SC216 24x2.5" + the cheap JBOD controller (no ipmi / fan controls / etc but also CHEAP), and then you can slowly add more SSD as you can afford/need... create a 'new' pool of mirrored VDEVs of SSD, start with 1 more SSD like you have now if you want, add a SLOG later if needed.

- PCIE HBA w/External Ports
- Supermicro SC216 Chassis ($200-300 depending on PSU, backplane, etc.)
- 2nd SSD to start your new pool at minimum, 3x ssd to do a stripe in there

Then, as you need more capacity and performance add additional 2x SSD Vdev to your new VM pool.

Of course, the cheapest and simplest would be to add a PCIE device for fast SLOG and use your existing SSD for L2ARC and your existing pool for capacity :) but then you're limited again going fowrard.
 
M

modder man

Active Member
Jan 19, 2015
657
84
28
32
Right a 2U JBOD is actually what i would like to do or go up to one of the 24/36 bay supermicro boxes. The problem is that either of those changes require I pay the DC more per month which I am looking to avoid. Typically I spend way to much money on this hobby but I do have a hard time justifying increasing monthly expenses.

Sounds like for now I need to throw a SLOG and l2arc at it and do some testing. I have an intel desktop ssd I could throw in as the L2arc for now if it even needs it. I could give the box up to probably 64GB of ram if it makes use of it. I was also looking into installing Perix's FVP on the host to use some ram as a cache host side.
 
T_Minus

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,640
2,057
113
If you're going to use it for VMs with SLOG and L2ARC then I think you for sure want to dedicate at-least 16GB of RAM to the ZFS VM, and see what utilization is like (ARC stats) you may need to go up to 24 or 32 depending on VM usage. But, that should hopefully be cheap (RAM).
 
M

modder man

Active Member
Jan 19, 2015
657
84
28
32
The host already has 192GB in it. At this time I only use about 86GB, so I am ok with allocating ram if it will benefit the performance.
 
M

modder man

Active Member
Jan 19, 2015
657
84
28
32
I really appreciate all of the effort you both put into the discussion here. It is something I just havent been able to play with enough as I dont have any more pieces laying around to experiment with.
 
M

Marsh

Moderator
May 12, 2013
2,645
1,496
113
Today, I picked up 2 x 640gb Fusion IO MLC card for $125 each.
1.2 TB cheap Fusion IO would fixed your VM problem.
 
You must log in or register to reply here.
Top Bottom