What did I miss? Intel S2600WT2

AJXCR

Active Member
Jan 20, 2017
565
95
28
32
Due to an extremely good deal, the NVMe drives I'm planning to initially populate this machine with are m.2 960GB PM953's. Relative to newer NVMe drives these are quite slow, but they include PLP and for the time being I can buy them cheaper in bulk than PM853's (about $0.22/GB)... Which is attractive.

The one performance metric that is way out of line is random write IOPs which is only 19K (less than 1/5 of a standard SATA3 850 Pro). Anyone want to talk me out of them?

The alternative would probably be Intel 750's as I have 4x 1.2TB drives still in the box and for the time being I'd be limited to a total of 8. 8TB of NVMe flash (4TB usable mirrored) @$0.22/GB is tough to pass up..
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,046
1,583
113
CA
@AJXCR I don't know about the PM953 but the 850 PRO is a consumer SATA drive, you never should compare enterprise drives to consumer. As I said I don't know about the PM953 so this is just made up # examples to illustrate. The PM953 is rated at 19,000 IOPs for 100% of disk the 850 Pro is likely 8GB, and the PM953 may go from 19k to 10K IOPs once it hits steady state where as the 850 Pro drops below 10K and stays there even though it's rated for what 80k? Also, NVME drive are not limited by the SATA QD so you'll get more performance no matter what from NVME if you have the work load for it.

This is true for Intel 750 NVME as well as pretty much all consumer drives, and why so many of us here are on the hunt for enterprise drives on ebay :)

When you're looking at the Intel PDF specs for their various NVME drives be sure to checkout the "mixed workload" performance you'll see the P3700 greatly outperforms the other models when there are writes and reads going on at once and this is by a very large margin.

Will 4x1.2TB 750 NVME work for you... maybe, I don't know the work load. BUT, just remember you're likely never going to get anywhere near the rated performance of a consumer drive because they're tested differently, and run different firmware as to not jeopardize their enterprise sales.
 

AJXCR

Active Member
Jan 20, 2017
565
95
28
32
@AJXCR I don't know about the PM953 but the 850 PRO is a consumer SATA drive, you never should compare enterprise drives to consumer. As I said I don't know about the PM953 so this is just made up # examples to illustrate. The PM953 is rated at 19,000 IOPs for 100% of disk the 850 Pro is likely 8GB, and the PM953 may go from 19k to 10K IOPs once it hits steady state where as the 850 Pro drops below 10K and stays there even though it's rated for what 80k? Also, NVME drive are not limited by the SATA QD so you'll get more performance no matter what from NVME if you have the work load for it.

This is true for Intel 750 NVME as well as pretty much all consumer drives, and why so many of us here are on the hunt for enterprise drives on ebay :)

When you're looking at the Intel PDF specs for their various NVME drives be sure to checkout the "mixed workload" performance you'll see the P3700 greatly outperforms the other models when there are writes and reads going on at once and this is by a very large margin.

Will 4x1.2TB 750 NVME work for you... maybe, I don't know the work load. BUT, just remember you're likely never going to get anywhere near the rated performance of a consumer drive because they're tested differently, and run different firmware as to not jeopardize their enterprise sales.
If I went with the Intel 750's I would buy an additional 4 drives for a total of 8.

Understood on enterprise vs consumer products. I used the 850 Pro as an example simply due to fact that it is a staple consumer level ssd.

Based on what I've read, the PM953 is a fairly dated low end prosumer drive. Specs are:
Form Factor: 2.5”/m.2
Capacity: 480GB, 960GB, and 1.92TB
Host Interface: PCIe Gen 3.0 x4 @ 32 Gb/s
MTBF: 2,000,000 hours
Power Consumption (Active/Idle): 9W / 1.9W
Endurance: 1.3 DWPD for 3 Years (0.8 DWPD for 5 Years ) >>> Not great
UBER: in 10^17
Random Read (4K): Up to 240,000 IOPS
Random Write (4K): Up to 19,000 IOPS
Sequential Read: Up to 1,000 MB/s
Sequential Write: Up to 870 MB/s
Physical Dimensions: 70 x 100 x 7 mm
Weight: Up to 74 grams

Specs on a high level enterprise drive from Samsung drive from the same era (PM1725) are:
Form Factor: 2.5”
Capacity (GB): 800GB, 1.6TB, 3.2TB
Host Interface: PCIe Gen 3.0 x4
MTBF: 2,000,000 hours
Power Consumption (Active/Idle): 25W/7W
Endurance: 5 DWPD (5 years)
Sequential Read: Up to 3,100 MB/s
Sequential Write: Up to 2000 MB/s
Random Read: Up to 750,000 IOPS
Random Write: Up to 120,000 IOPS
Physical Dimensions: 69 x 100 x 14 mm
Weigh: 140 grams

While I recognize that these numbers are fairly stratospheric, 19K RW 4K seems on the PM953 seems low..

RW 4K for other common drives:
P3700: 90,000 IOPS
Intel 750 (8GB Span): 290,000 IOPS
Intel P3600 (100% Span): 23,000 IOPS

My go to review for the Intel 750 vs the P3700 has always been:
Intel's 750 Series solid-state drive reviewed
 
Last edited:

AJXCR

Active Member
Jan 20, 2017
565
95
28
32
Also, it's my understanding that in FreeNAS the SLOG device should outperform the pool.. with 4xPM953 write performance am I stuck looking at something like mirrored P4800x drives? ...Because that's out of the question based on current pricing. I could swing mirrored P3700's or PM1725's, but I'm not sure that would get me there with 8x PM953 read performance?
 

Rand__

Well-Known Member
Mar 6, 2014
4,592
912
113
CPUs:

Spicy ES CPU Sale
2667 QS - not sure if those are interesting
There was one person selling retail v3/v4s but couldn't find it just now

750 as pool - those do have limited durability, so it very much depends on the amount you're going to write.
Also, if your going ESX/nfs then slog will matter -
1. You can't join multiple devices for one faster slog in FreeNas
2. P3700 is a much better slog than 750 from my tests. Optane will outperform the P3700 by far aparently (Intel Optane: Hands-on Real World Benchmark and Test Results)

PM953s will throttle under usage quite quickly due to heat issues. Even if not you will need an slog with supercap as they don't have one. Lastly, they suck as slog (performance wise). If the are cheap enough, (try to) get a pool of them with proper cooling and add a (pair of) p3700 in front. Not sure about their endurance though
 

AJXCR

Active Member
Jan 20, 2017
565
95
28
32
CPUs:

Spicy ES CPU Sale
2667 QS - not sure if those are interesting
There was one person selling retail v3/v4s but couldn't find it just now

750 as pool - those do have limited durability, so it very much depends on the amount you're going to write.
Also, if your going ESX/nfs then slog will matter -
1. You can't join multiple devices for one faster slog in FreeNas
2. P3700 is a much better slog than 750 from my tests. Optane will outperform the P3700 by far aparently (Intel Optane: Hands-on Real World Benchmark and Test Results)

PM953s will throttle under usage quite quickly due to heat issues. Even if not you will need an slog with supercap as they don't have one. Lastly, they suck as slog (performance wise). If the are cheap enough, (try to) get a pool of them with proper cooling and add a (pair of) p3700 in front. Not sure about their endurance though
Thank you for the reply! A couple of clarifications:
-I was under the impression that the 2667's for sale in that thread were V1's? Looking for V4/V3 processors.
-When referencing 2xSLOG drives I was thinking more along the lines of mirrored for redundancy
-I think I can address the heat issues, but I suppose I won't know for sure until I get them in and test.
-Tell me more about this "supercap". The PM953's do incorporate PLP via "tantalum capacitors". See link below:
http://www.samsung.com/us/dell/pdfs/samsung_flyer_PM953_v2.pdf
-No interest in running the 953's as SLOG devices... The idea was to run 8, 12, 14, etc as the primary storage pool.. This is what I was getting at with the SLOG question. It's my understanding that the SLOG device should outperform the storage pool it's supporting. Although the PM953 is an entry level enterprise drive, I expected that with 8 to 14+ (once the new NVMe capable HBA's come out) running in a pool, they might out perform everything but Optane..
 

Rand__

Well-Known Member
Mar 6, 2014
4,592
912
113
-The QS ones are v4 "2x E5-2667v4 (QKF0) - 8 Cores, 16 Threads, 3.2-3.6ghz, 25mb Cache, 135w - $450 Per Shipped"
-2xSLOG, ok , sounded like speed up idea
-Supercap - hm maybe I mixed those up then - was referring to the m.2 ones which where particularly popular when the 750 came out and at least the m.2 did not have plp that I knew of.

outperform - depends on your workload again - a single user will most likely have better speed with an p3700 or optane, many user will benefit from the mutitude of disks, so total might be higher.
 

AJXCR

Active Member
Jan 20, 2017
565
95
28
32
-The QS ones are v4 "2x E5-2667v4 (QKF0) - 8 Cores, 16 Threads, 3.2-3.6ghz, 25mb Cache, 135w - $450 Per Shipped"
-2xSLOG, ok , sounded like speed up idea
-Supercap - hm maybe I mixed those up then - was referring to the m.2 ones which where particularly popular when the 750 came out and at least the m.2 did not have plp that I knew of.

outperform - depends on your workload again - a single user will most likely have better speed with an p3700 or optane, many user will benefit from the mutitude of disks, so total might be higher.
Looks like you were correct.. I've been searching for a pair of QS 2667V4's, 2687WV4's, or 2690V4's for quite some time and would have snapped these up immediately. Unfortunately it looks like they've already been sold.

On the drives, I believe the non enterprise m.2 drives do not incorporate PLP, but the PM953's & PM963's both include PLP via tantalum capacitors in the m.2 form factor.

I don't follow the last statement... Would the P3700 or Optane SLOG device for a large NVMe primary storage pool not serve a both a single user and a large number of users well? As long as we're talking about NVMe primary storage + P3700 or Optane SLOG, it would seem like a large NVMe array would overrun even the high end NVMe drives used as SLOG.
 

Rand__

Well-Known Member
Mar 6, 2014
4,592
912
113
Not sure where you are looking at but in @ShepsCrook 's thread here (Spicy ES CPU Sale) they still look available:)

Yes but only at a larger number or users - if you have a single user (single thread, QD=1 or small), then a single fast drive wins over n slower drives.
At higher users (QD>32 so to speak, when a single P3700 is saturated) the n drives will be faster.
Dont forget, its not only IOPs and write speed but also latency we're talking about when considering SLOGs, and I very much doubt many PM863s have a better latency than a p3700
 
Last edited:
  • Like
Reactions: T_Minus

gigatexal

I'm here to learn
Nov 25, 2012
2,766
529
113
Portland, Oregon
alexandarnarayan.com
Why are you mirroring the drives? Mirrored 4800x's come on dude. Unless you're getting hardware that fell off a truck this all seems way overkill. Wouldn't pedestrian SSDs saturate a 10GB link given that most of the hot files will be cached in ram and served from ram?
 

Rand__

Well-Known Member
Mar 6, 2014
4,592
912
113
Mirror'ed slog just for fault tolerance :)

Edit: Picked up one of those too ... hope my ES CPUs work :p
 
Last edited:

AJXCR

Active Member
Jan 20, 2017
565
95
28
32
Not sure where you are looking at but in @ShepsCrook 's thread here (Spicy ES CPU Sale) they still look available:)

Yes but only at a larger number or users - if you have a single user (single thread, QD=1 or small), then a single fast drive wins over n slower drives.
At higher users (QD>32 so to speak, when a single P3700 is saturated) the n drives will be faster.
Dont forget, its not only IOPs and write speed but also latency we're talking about when considering SLOGs, and I very much doubt many PM863s have a better latency than a p3700
Next paragraph down under the SOLD: heading the 2667V4's are listed..

Maybe I'm making a mistake.. Although for a pool of disks read/write speeds would be improved, would latency not be affected? Note that the drives are PM953's not PM863's. For example, lets say I have 8x PM963's in RAID0, write speeds go up, read speeds go up, what happens to latency?
 

Rand__

Well-Known Member
Mar 6, 2014
4,592
912
113
Some are sold, some are available - thats how I read this :) Best simply ask him :)

Latency will stay the same I'd imagine, I think thats a metric inherent to the drive and QD...
 

AJXCR

Active Member
Jan 20, 2017
565
95
28
32
Some are sold, some are available - thats how I read this :) Best simply ask him :)

Latency will stay the same I'd imagine, I think thats a metric inherent to the drive and QD...
Good point.. PM sent. I was thrown off by the 2x E5-2667V4 specification at the top, and then sold listing at the bottom.
 

AJXCR

Active Member
Jan 20, 2017
565
95
28
32
Why are you mirroring the drives? Mirrored 4800x's come on dude. Unless you're getting hardware that fell off a truck this all seems way overkill. Wouldn't pedestrian SSDs saturate a 10GB link given that most of the hot files will be cached in ram and served from ram?
I like sports cars, super bikes, planes, HiFi, and expensive watches as well.... If you ain't first you're last.

All joking aside, I was under the impression that mirrored ZIL devices was common practice, if not a requirement:
A Complete Guide to FreeNAS Hardware Design, Part III: Pools, Performance, and Cache - FreeNAS - Open Source Storage Operating System
Building new DemoEval Lab Storage with FreeNAS

Either way, if this test system works as intended and ultimately ends up serving corporate duty with my corporate data, who cares about an extra P4800X if it helps mitigate the risk of losing an entire pool of data.

That being said, do I currently want to buy two (or even one) P4800X drive for a test box? No. If it proves necessary in a production box? ....Where do I sign?

Network will be 40G.
 
Last edited:

Rand__

Well-Known Member
Mar 6, 2014
4,592
912
113
Requirement - no.
Best practice - depends on the importance of the data. If you can't afford data loss at all (and have everything mirror'ed) then o/c you'll mirror slog as well.
For a test box - no need unless you explicitly want to test that scenario
 

AJXCR

Active Member
Jan 20, 2017
565
95
28
32
Making a little progress.. Next big group of packages scheduled to arrive tomorrow: