SSD choice: 750 vs 850 Evo's vs P3600 for multiple OS disk

[Ramos]

I have a dual 2670 rig that I plan on using for virtual clusters for Hadoop and other cluster software tests with up to 6 VM's running concurrently, maybe 7 or 8. I plan on splitting the drive into virtual datadrives too for the cluster nodes so I can use the SSD to its full potential before investing in more SSD space for bigger trials. I don't expect to be hammering all nodes at all times on disc IOPS but for ETL it would be supernice to have a FAST ssd, IOPS wise.

I prefer to keep the machine off irondisc stuff for active data trials.

Atm 1 TB distributed storage is enough using the Reddit table for most distributed stuff.

For that, (and for 4k/5k video editing down the line with a Kinemax 5k) I need a powerful SSD base to drive the OS's.

Budget 700 euro isch.

Right now I found the following locally, (EU)

- Intel P3600 800 GB €363
Pro: Fast IOPS reads, enterprise TBW of 4380 TB
Con: Bad write IOPS of just 50k random 4k. Might be okay for video editing but not sure with the poor write IOPS.

- Intel 750 1.2 TB €709
Pro: Superb IOPS in both, good space for stuff
Con: BIG con in just a feeble 219 TB TBW. Such a killer, especially for the video stuff.

- 850 Evo's 500/1000 GB.
Pro: "Cheap" and reliable. With HBA I can easily go to 4 TB total space on SSDs.
Con: Only 2 Sata 6 gbps ports and thus a €130 HBA comes on top for more than 2. Which means I either go 2 x Big TB or lots of cheapos and a HBA.

So far own conclusion:
- I would like optimally a 750 that doesn't quit right after it got started.
- A Hybrid of a P3600 and then 2x 500 GB 850 Evo's at 133 euro is right around 700.
- No idea what used stuff would be useful, but iron disks are out of the picture since I need simul speed for at least 4-6 VM's at a time, and used SSDs sounds like tampered TBWs and near dead drives, but I dunno that market.

 

[T_Minus]

The p3600 has > write iops than all other drives you mentioned... don't be confused by the consumer "write iops rating". It also has > mixed workload iops than the others too.

I would NEVER Use 850 Evo.

 

[Ramos]

I forgot to mention (or think about in the OP) that there is no way I can afford Intel 750/3x00 for 4 TB of storage (distributed requires 3 copies of data, so for overhead for 1 TB of actual data, I need 3-4 TB), so I need a 850 Evo setup as well as the 6-8 x OS drive. Drive ports are limited so need to go fairly big or HBA (ie IBM 1015 ie pricy).

OS drive,

Which metrics are you judging from?

According to the official,
http://www.intel.com/content/dam/ww.../product-specifications/ssd-dc-p3600-spec.pdf
(p.10, 800 GB column)
The P3600 has 110k/55k r/w random 4k 70/30 IOPS. Sounds awfully low.

The P3600 might be out of contention again, the price listed was half of everyone else's price and not deliverable till 1-3 months, so I smell a typo or bad batch. Not paying 700 euro for an 800 GB semi outdated SSD. Seems I need to get an Intel 750 and just replace it every 2 years or so. Pretty pricy endeavour.

The P3500/P3600's are getting dated. Is there a used market for these? ... Was thinking that with a TBW that high, used could come into the picture again.

Data drives,

I don't see why I shouldn't use the 840/850 Evo's, they have been rock solid for me and friends for years as the defacto best cheap SSD drive there is, since Intel dropped the lead. Sure they can break once in a full moon, but toss it and flip in another.

Are they that bad or are you talking 24-7 applications?

Or some other cheaper SSD for that. Still only got 700 euro for it max though. Any (real) alternatives to the 850 Evo's?

Looks like OCZ Trion 960GB's might be relevant,
Solid State Drives (SSDs) mit Kapazität ab 400GB, Schnittstelle: SATA, IOPS 4K lesen: ab 75k IOPS, IOPS 4K schreiben: ab 50k IOPS, TBW: ab 200TB, Gelistet seit: ab 2016 Preisvergleich | Geizhals Deutschland

Procurement plan so far,

- Right now plan: (Compromise and go 2TB storage)
OS: Intel 750 400 GB, €300
Data: 2x OCZ Trion 150 960 GB, 2x €225 (on the 2xSata 6 Gbps onboard ports)
750 euro. Will have to go down to just 500 GB distributed datasets but that's really okay, it's not a real cluster anyway and I have those at work.

- Long term expandable plan: (8x480GB for 4TB storage, down the line, add €600 later)
OS: Intel 750 400 GB, €300
Data: 1x IBM Lenovo N2225 HBA + 4x Mushkin Triactor 480 GB, €88 + 4x €100
788 euro. Will have to go down to just 500 GB distributed datasets but that's really okay, it's not a real cluster anyway and I have those at work.

Any one got some better setups using only SSDs? .. Used ones with enterprise TBW numbers are okay but if US I need to add 25% toll and probably 25-50 USD shipping.

 

[BackupProphet]

OCZ Trion is awfully slow for data. I highly recommend getting something else, infact avoid consumer drivers. Get these instead Samsung PM863 2.5" 480GB SATA OEM Enterprise DC SSD

Performance is on a completely different level.

 

[Evan]

PM863 is a solid cheap Sata Ssd.
And they come upto 3.84tb in size (granted that is a $2000 drive though but still good value for performance with a warranty)

Smaller sizes are generally also reasonably available on eBay etc

 

[Ramos]

Editted heavily, sorry misunderstood two links and saw the provided link WAS for used PM863s.

OS Disk:

The P3600 came back into cheap price and I snatched the last one I think.

Does anyone have any experience with P3500 / P3600 / P3700s on S2600CP motherboards and booting from them?
(Assuming newest BIOS of 02.06.0002 from May 2016)

Data disks:
Okay, PM863s are okay, but if I go for those, a SM863 seems to be like 10% more (from new, where I roughly live) so aren't those a better choice with an insane TBW and 5 yr warrenty over 3 yr warrenties for PM863's?

I might have to compromise my budget quite a lot it seems.

 

[Evan]

I have no experience with the SM863 drives but yes would assume they are much better and new not a great deal of difference in price but generally availability of the PM863's in the market place is greater.

 

[Deslok]

What about the 1tb mushkin reactor drives? they're cheaper than the evo and are MLC , they lose a bit of top end performance but should be more predictable

also i wouldnt worry about killing the 750 at ~200tb, it's a warranty limitation to stop you from using it in enterprise environment it should live for much more than that(most drives outlive their warranty number take a look here The SSD Endurance Experiment: They're all dead )

 

[Ramos]

OS Disk:
I have ordered the P3600 now, but it will probably be cancelled to wrong pricing,

And even with JD's god-like thread on the subject,...
NVMe on Intel S2600CP
... it seems to be a luck based nightmare to even get half-arsed performance. But it might be worth a shot and if the card comes through I can try it out. Also need a good gfx card later on, so pretty much need those ports to go Pci 3.0 speeds anyway.

Data disks:

The Mushkin Reactor looks fairly okay but noone has numbers on TBW so that worries me a great deal.

The PM863's are still the prime target for now, but every damn SSD is now as dependent on that BIOS hack working, as I need more than 2 total and 3 Gbps ports are not an option.

Even a bloody stoneage-simple HBA is hard to find at a low price that is also on that list for the 3.0 8x I need. All of the serial number products I looked at were HW raid cards, which I consider obsolete with FreeNAS around. If the controller goes, all is gone and they are too expensive to just use as JBOD HBAs for just 4 SSDs. Cheapest I saw, delivered here, via US ebay or otherwise, including taxing was north of 200 usd.

And I probably need more than the "safe" 2.0 8x safety speed the slot defaults to with Intels BIOS mess. Even the 3.0 16x slot will throttle to 2.0 8x.

Alternatively I need to get 2 SSDs so big the entire budget goes out the window and I can't afford that.
2+ TB SSD seems to be sponsored people only and I don't get paid for this testing this, it's just for experiments for work. Iron disks will be so slow that the whole idea will defeat itself.

Conclusion right now:

Pretty bleak. Frustrated to bits atm. I can't sell the rig if the BIOS show goes south.

Options,
- Go all-in and get that BIOS shit done, risking the board dies. If it does, get a 340 euro new Asus board or sell out and cut loses, if that makes more sense and basically go for option 2 with a smaller sales value.

- Sell the whole rig and just get a minimum rig for spare time and remoting to train on AWS and somehow bill someone else for the training. Bring-your-own-license is apparantly an option, but AWS are notoriously hard to get a partner deal out of (APN) so one can train for free for X monies a month. So might be more expensive in the long run, especially if they won't accept the Partnerworld licenses.

Edit: Removed some colorful frustration.

 

[Deslok]

Anandtech put the endurance at 144TB during their review Mushkin Reactor 1TB SSD Review
But an interesting note is it uses the same NAND (and controller) as the crucial BX100 which listed 72TB across the board Crucial BX100 (120GB, 250GB, 500GB & 1TB) SSD Review
Assuming nothing other than an increase in die count the BX100 should have seen the 1TB model (which is roughly 8x larger than the 120gb model) have a 576TB spec but that would be enterprise territory, endurance ratings really are arbitrary unfortunately, they're picked for the warranty period to avoid people using drives in ways they don't want.

Also i don't know why we're suggesting the PM863 it's a TLC drive just like the 850 evo they should perform the same other than the presence of PLP although I'm not sure that's relevant for this use case. HBA wise the IBM M1015 is on ebay regularly in the 75-100 range and can be flashed to IT firmware, I'm not sure on it's UK pricing.

 

[T_Minus]

Anandtech put the endurance at 144TB during their review Mushkin Reactor 1TB SSD Review
But an interesting note is it uses the same NAND (and controller) as the crucial BX100 which listed 72TB across the board Crucial BX100 (120GB, 250GB, 500GB & 1TB) SSD Review
Assuming nothing other than an increase in die count the BX100 should have seen the 1TB model (which is roughly 8x larger than the 120gb model) have a 576TB spec but that would be enterprise territory, endurance ratings really are arbitrary unfortunately, they're picked for the warranty period to avoid people using drives in ways they don't want.

Also i don't know why we're suggesting the PM863 it's a TLC drive just like the 850 evo they should perform the same other than the presence of PLP although I'm not sure that's relevant for this use case. HBA wise the IBM M1015 is on ebay regularly in the 75-100 range and can be flashed to IT firmware, I'm not sure on it's UK pricing.

I don't know enough about the PM836 and 850 EVO off the top of my head to compare the 2 drives... but just because they both have TLC or drives both have MLC doesn't mean they're the same firmware, same performance, etc.... I would not compare drives like that. (Note: I'm not saying those two aren't similar/same performance but in general it's not a good way to compare.)

There are a lot of threads about asking about SSD, NVME, etc... and I've said the same thing in all of them, I suggest searching the forum and reading the many other topics to understand that event though IOPs = IOPs there is a difference in consumer work load performance testing and enterprise sustained / VM work loads.

A quick example is some consumer SSD rated for 85,000 IOPs and test that... but when you use them more than a few minutes the IOPs drop SIGNIFICANTLY, and then a few more minutes the IOPs are down to 900-1200 !!! Where-as the enterprise SSD will drop, level out, and then might climb a small bit and keep steady there.

An array of Intel S3500 for OS drives is really hard to beat cost-wise and performance wise. You're looking at $90-100 for 300GB, 180-200 for 600GB. S3700 400GB for 170$ right now is a kick-ass drive for the price too!

As you noted the P3600 write IOPs are not special for NVME, and really no other drive (other than the HGST SAS) are special as the S3700 SATA and Intel P3700 NVME. If you NEED write-iops then you need to pay to play. The P3700 NVME smokes all other NVME drives in mixed workload (by a large %) but you pay $1/GB or more. P3600/P3605 is a great 'capacity' NVME drive, and good for running a lot of VMs from due to the 50c/GB pricing.

There are some P3700 deals on ebay but the price per-GB is more $$ but they'll absolutely blow youa way running numerous VMs!!

 

[Ramos]

Yeah I was gunning for the 1015 too as I consider it the HBA standard (ie I need a good reason to pick something else).

But then I realized it is just Pci-e 2.0 8x natively and thus 30 isch Gbps combined and that it will becomes a bottle neck way before 8x PM863's or similar.

Either way I still need to BIOS lottery the board this weekend and see where I am at and then act based on that.

 

[Deslok]

I don't know enough about the PM836 and 850 EVO off the top of my head to compare the 2 drives... but just because they both have TLC or drives both have MLC doesn't mean they're the same firmware, same performance, etc.... I would not compare drives like that. (Note: I'm not saying those two aren't similar/same performance but in general it's not a good way to compare.)

There are a lot of threads about asking about SSD, NVME, etc... and I've said the same thing in all of them, I suggest searching the forum and reading the many other topics to understand that event though IOPs = IOPs there is a difference in consumer work load performance testing and enterprise sustained / VM work loads.

A quick example is some consumer SSD rated for 85,000 IOPs and test that... but when you use them more than a few minutes the IOPs drop SIGNIFICANTLY, and then a few more minutes the IOPs are down to 900-1200 !!! Where-as the enterprise SSD will drop, level out, and then might climb a small bit and keep steady there.

An array of Intel S3500 for OS drives is really hard to beat cost-wise and performance wise. You're looking at $90-100 for 300GB, 180-200 for 600GB. S3700 400GB for 170$ right now is a kick-ass drive for the price too!

As you noted the P3600 write IOPs are not special for NVME, and really no other drive (other than the HGST SAS) are special as the S3700 SATA and Intel P3700 NVME. If you NEED write-iops then you need to pay to play. The P3700 NVME smokes all other NVME drives in mixed workload (by a large %) but you pay $1/GB or more. P3600/P3605 is a great 'capacity' NVME drive, and good for running a lot of VMs from due to the 50c/GB pricing.

There are some P3700 deals on ebay but the price per-GB is more $$ but they'll absolutely blow youa way running numerous VMs!!

In regards to the Samsung drives they're using the same nand and the same (2tb evo) or related(other sizes) controller as the pm863 there will be tuning and firmware tweaks but if the pm863 is ok for an application an equal sized and adjusted(over provisioned) evo should perform similarly outside of plp

 

[Ramos]

Thanks @T_Minus , great post.

I kinda knew the consumer drives would tank after a while, but not to 900-1200, damn.

Well, I guess the good news is that
- The BIOS lottery will have to be done for me, no matter what. Its just an all-or-bust solution and I hate facing those with my own wallet on the line. Not at all blaming anyone but Intel btw, it's just ... Ugh, why does a policy stop me, so stupid.

So assuming I get the BIOS sorted, the P3600 is coming anyway. It's AMAZING for reads and since it will run the main OS and about 8 VMs, mostly concurrently I hope, it will be fine for smaller writes consistently.

The data work in Hadoop / distributed (GPFS-FPO etc) terms will of cause have a good amount of writes, due to moving data around and the stupid amount of logging that goes on, but I will try and spread that over a bunch of PM863's unless something else comes up.

I looked and maybe its ebay's friendly calculator (that adds shipping and import tax from the US) to my prices, but I cannot find those items you mention in your post. The PM863s, if I buy 4 in a bunch, at 129 usd each, still goes to 158 usd a piece once I have them in my hand, so its pretty significant.

Atm I think "plain" SSDs with big TBWs will be fine and at 30c/GB the PM863s are fine. My P3600 is Pci-e 800 GB and came at 390 euro shipped so just over 50c/GB.

I'll deffo more after Intel P3x00 alternatives as NVMe based ones if they are available that cheap.

Btw, a manager at work told me there is a special HP? card that goes into a Pci 3.0 16x slot and provides 4 slots (HBA like) of NVMe ports for Pci 3.0 4x full-speed drives. He gives around 4k/600 usd for it at the company with rebate, but I have no idea what card it is and googling for it came up empty.

I'll try and ask if I see him and we both have time tomorrow.

 

[Ramos]

Btw, a manager at work told me there is a special HP? card that goes into a Pci 3.0 16x slot and provides 4 slots (HBA like) of NVMe ports for Pci 3.0 4x full-speed drives. He gives around 4k/600 usd for it at the company with rebate, but I have no idea what card it is and googling for it came up empty.

I'll try and ask if I see him and we both have time tomorrow.

Found it. Again, right on StH, or well he found it here too,
4 solutions tested: Add 2.5" SFF NVMe to your current system

Looks awesome, if the BIOS fix will work, I might borrow one for testing. And reasonable price.