NVME Drive for VM Storage

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

JDM

Member
Jun 25, 2016
44
22
8
33
Preparing to build a VM server running Proxmox here in the nearish future (hopefully next 1-2 months). Trying to decide which NVME drives to use in the build. I'll have some rotational drives for bulk storage for the VM's, but for the actual OS images was planning on putting in 4 Intel NVME SSDs (striped mirrors in ZFS), but may need start with just two and add the other two later.

There are a lot of options out there, the Intel 750's looked good initially, but reading some comments here on STH forums they may not behave well for VM workloads. Looking at the P3600's and P3700's on ebay, seems like folks were getting better deals on them earlier this year, but there are some good prices that pop up from time to time. Intel has been rolling out some newer drives recently also curious if any of those may be better? The P3600's and P3700's have been out a while.
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,625
2,043
113
Just got done with a new build actually myself with proxmox and NVME. Right now I loaded it with 4x SAS3 and 2x P3600 NVME with the capacity to add 2 more 2.5" (SAS3 or NVME). Plan is to run it like this and see what drives I want to add/remove for my needs.

The p3605 oracle oem are the cheapest but no warranty.
 

( )

New Member
Jul 8, 2017
27
4
3
Preparing to build a VM server ... Trying to decide which NVME drives to use in the build. ... for the actual OS images was planning on putting in 4 Intel NVME SSDs (striped mirrors in ZFS), but may need start with just two and add the other two later.

... may not behave well for VM time to time.
ZFS = my favorite.

I like to use a 'Benchmark / Calculator Website' that takes user Benchmarks and lets you sort and compare.

Here is Intel's best vs. Samsung's best: UserBenchmark: Intel DC P3700 Series NVMe PCIe vs Samsung 960 Pro M.2 - note how one wins in every way, for less than half the cost.

That's a fairly full featured 'B/C Site' but find a better one and post a Link for us if you can.

I like to sort on Write Speed and find the highest ranked. Capacity depends upon number of users and how much ZFS Features you want enabled.

After choosing a Drive be certain to read 'ZFS specific Reviews' for that particular Drive to ensure it works properly for the flavor and version of ZFS you are using in addition to the OS/Hypervisor (and in some cases what is to run inside the VMs). Don't get an excellent Drive but have Driver difficulties or the need to use a non-optimal Driver vs. one that will wring out all the performance.
 

i386

Well-Known Member
Mar 18, 2016
4,220
1,540
113
34
Germany
Here is Intel's best vs. Samsung's best: UserBenchmark: Intel DC P3700 Series NVMe PCIe vs Samsung 960 Pro M.2 - note how one wins in every way, for less than half the cost.
I wouldn't buy based on these "benchmarks". The 960 pro is not an enterprise drive and drops in performance once the (volatile) cache is filled up and the trim or equivalent optimization process can't kick in.
This link shows the performance drops in sustained workloads: Samsung 960 Pro vs 950 Pro Performance & Thermal Throttling
 

( )

New Member
Jul 8, 2017
27
4
3
Throughout JDM's Post he made references to saving money: from free Software to buying fewer Drives, even waiting for a deal on EBay.

I clearly indicated he need not choose those particular Drives or rely upon that Site, providing it as an example only. There's many comparisons available on that specific Site and others on other Sites, better links help us all.

I quickly skimmed your Link and thoroughly read the Conclusion including two of the related Links, this one in particular: Samsung 950 Pro M.2 Additional Cooling Testing .

With all that he has been provided I trust in him that he will be able to make his own decision. Even more assistance is offered if he's still in need.

Thanks for the additional information you provided.
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,625
2,043
113
I wouldn't buy based on these "benchmarks". The 960 pro is not an enterprise drive and drops in performance once the (volatile) cache is filled up and the trim or equivalent optimization process can't kick in.
This link shows the performance drops in sustained workloads: Samsung 960 Pro vs 950 Pro Performance & Thermal Throttling
This 1000000%.
Do not rely on that site for anything server /enterprise related to SSD or NVME.

Throughout JDM's Post he made references to saving money: from free Software to buying fewer Drives, even waiting for a deal on EBay.

I clearly indicated he need not choose those particular Drives or rely upon that Site, providing it as an example only. There's many comparisons available on that specific Site and others on other Sites, better links help us all.

I quickly skimmed your Link and thoroughly read the Conclusion including two of the related Links, this one in particular: Samsung 950 Pro M.2 Additional Cooling Testing .

With all that he has been provided I trust in him that he will be able to make his own decision. Even more assistance is offered if he's still in need.

Thanks for the additional information you provided.
Practically this entire forum user base 'waits on ebay' for deals that's not an indicator you should give him bad advise, and then hope he's going to do his own research on something further he chose based on a site you gave him. He started the post asking for advise/suggestions not asking to be shuffled off to a 3rd-party review site made for consumer SSD.
 
  • Like
Reactions: james23

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
Depending on the amount of spinners you have (and speed requirements) you could get away with having a pair of P3700's (400gb) as slog instead of a dedicated OS drive pool to save some money.
 

Tom5051

Active Member
Jan 18, 2017
359
79
28
46
I have 950's and 960's in my ESXi hosts and they are very fast and give my Intel 750 a run for it's money. I have heat sinks on the drives to this prevents any sort of thermal throttling. The hottest they ever get during sustained read / writes is 35 celcius. Without heat sink the 950 will throttle after about 60 seconds and the 960 after about 135 seconds.
 
  • Like
Reactions: ( ) and Pepe

dswartz

Active Member
Jul 14, 2011
610
79
28
I have 950's and 960's in my ESXi hosts and they are very fast and give my Intel 750 a run for it's money. I have heat sinks on the drives to this prevents any sort of thermal throttling. The hottest they ever get during sustained read / writes is 35 celcius. Without heat sink the 950 will throttle after about 60 seconds and the 960 after about 135 seconds.
What heat sinks are you using?
 

JDM

Member
Jun 25, 2016
44
22
8
33
Just got done with a new build actually myself with proxmox and NVME. Right now I loaded it with 4x SAS3 and 2x P3600 NVME with the capacity to add 2 more 2.5" (SAS3 or NVME). Plan is to run it like this and see what drives I want to add/remove for my needs.

The p3605 oracle oem are the cheapest but no warranty.
Sorry for the delay in responding to this thread, was busy moving from apartment to house (fun network projects to come, but that's for another time/post).

Thanks for the input on your build, look forward to finding out what you run into. I found the graph you linked to earlier this year highlighting the Intel 750's random 4K write performance (or lack there off). From other reading I've done it seems as though the 750's were a consumerized version of the P3600 which seems to make sense from that graph you show. Intel benchmarks for the 4K random IOPS are in ARK are different between the two models (8GB vs 100% span), with the test the graph is coming from showing what the 750's IOPS would be if tested in the same manner. On that basis, the 750 hits a constant ~22,000 IOPS vs. 30,000 for the P3600. With the extreme lack of availability of the 400GB P3600 on eBay or elsewhere and the plethora of 750's listed, I'm beginning to wonder if the 8,000 IOPS/drive is worth the hassle or price of finding multiple P3600's.

My mind isn't made up yet, as until I have the money fully ready I'm going to keep my eBay searches running and will continue pursuing other sources. With the P3520 coming in close to the same performance bracket for the 1.2TB version, and costing $630 new, I'm wondering if I can just get by with two of those and calling it good. Lots of options...
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,625
2,043
113
If the P3520 new is an option, do that :)

If you're spending $630 though don't' rule out the P3605 (P3600) 1.6TB fort he same price ;)
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
I used to get lightly used 1.2 TB disks (4 750s, one 3500) for sub €500 - original intel so full warranty.
Havent checked ebay on those in the last time though