Help selecting NVMe drives to finish build

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Cutha

Member
Sep 24, 2016
75
7
8
50
Canada
Hi,

I could use some advice with my home build. I was going to pick up 6 more M.2 NVMe SSD’s and then I saw a post on here and I realized, once again, that I am probably about to make a bad bang for the buck decision.

tl;dr Are there data center pull’s of NVMe drives that are of good quality and speed? I am budgeting ~$120/ea (6 total), top priority is speed and then size. Price is a bit flexible and I already have 2 x 256GB MyDigital ones.

Going by the many posts I have read while lurking, similar to some others on here I started with a desktop PC and loaded it up with HDD’s to make a ‘server’. It become unmanageable and along the way I ended up with a 4 post rack and a real server. My current setup is based around a Norco case with 20x3TB in RAID6 and dual E5-2670 in an ASRock MB.

I am very attached to Windows Server 2016 and I would like to keep using it. I tested a hyper converged setup with two servers and I really liked it so I decided to move to a 4 node setup. My current plan is a 4 node cluster using Storage Spaces Direct with 3 tiers of storage and all I have left to purchase is a few 4TB drives and 6 NVMe SSD’s.

I find that I am constantly learning while reading through these forums and sometimes I learn that I spent my money unwisely. So, what I am hoping for is some feedback, especially with regards to the parts I have not ordered yet. The servers host VM’s and provide storage for my workstations and for the kids gaming machines. We have a QDR Infiniband network setup (that I put together almost entirely from info on this forum).

For this build I have, or is in transit:

04 x SuperMicro 2U 12 Bay (6027R-E1R12N) w/ E5-2670’s
12 x SAMSUNG SM843T 2.5" 480GB SATA 6.0Gb/s MLC Enterprise SSD
14 x 4TB HDD
02 x 256GB MyDigitalSSD NVMe w/PCIe cards

What I still need to purchase:

06 x M.2 NVMe ~250GB SSD’s and PCIe brackets
02 x 4TB HDD

I want to use the NVMe drives as cache, the SSD’s as Hot Storage and the HDD’s as Cold Storage. I believe that using dual parity with storage spaces direct my setup should be able to withstand 2 node failure. Of course storage efficiency isn’t that great with this setup. For the VM’s I will likely just use the SSD’s and deduplication.

Currently unallocated I have:

04 x 500GB 840/850 Evo’s
10 x 3TB HGST Ultrastar HDD’s
04 x 3TB WD RED’s
01 x 4U SuperMicro 45 Bay 3.5” SAS2
08 x 2TB various HDD’s
02 x Intel RES2CV360

I wanted to use the 3TB drives to provide more space but from what I understand it would limit the 4TB drives to only 3TB usable because the columns would spread out evenly.

Are there good used NVMe drives to be had on ebay? The MyDigital ones that I have use MLC and the speeds are ok and they are pretty consistent. I have a hate on for the current crop SSD’s that use a cache and advertise the cache speed as if it’s a steady state speed. How is this any different than the RAM on a HDD? I am looking to spend about $120 each and I need 6 or 8 of them. Speed is the highest priority and then capacity. I actually have 2 SM961’s, I like them but they are a bit too pricey for me to get another 6.
 

Biren78

Active Member
Jan 16, 2013
550
94
28
Did you buy the PCIe brackets yet? If not, I'd get ones that can support 22110 so you can get M.2 SSDs.

Here are some options for low hour DC pulls
(too big) *100 Hrs* Samsung PM953 MZ1LV960HCJH-000MU 960GB M.2 Enterprise SSD PCIe Gen3 x4 | eBay
(also too big) *NEW* SK Hynix PE3110 960GB (Almost 1TB) Enterprise M.2 PCIe Gen3 SSD 3D NAND V2 | eBay

(maybe if you buy 4 they'll deal) *NEW* Samsung SM953 MZ1WV480HCGL-00003 480GB M.2 SSD PCIe 3.0x4 Gen3
(too expensive but maybe for a 5th) *NEW* Samsung SM953 MZ1WV480HCGL-00003 480GB M.2 SSD PCIe 3.0x4 Gen3

The MD ones they'll be better because they're smaller and so cheaper.

All of the M.2 and PCIE enterprise drives are 350-400GB+

If you don't care about power loss I'd still get Samsungs like this Samsung SSD PM961 256GB NVMe M.2 Internal OEM MZVLW256HEHP (960 OEM Bare Drive) | eBay
 
  • Like
Reactions: Cutha

Cutha

Member
Sep 24, 2016
75
7
8
50
Canada
Thanks for the reply guys!

I think with CSV I am ok without PLP and if the cluster doesn't protect me in that way please let me know.

I am going to post a WTB for the NVMe drives and see what pops up on here. The SM961's are on sale.

I have 4 of these PCIe brackets with heat sinks, they work well but the HS is a bit low and can hit the MB components.
 

Logan

Member
Feb 22, 2017
65
9
8
Saw in your other thread that you are looking for good performance at low queue depth. For high speed at low queue depth, if capacity isn't very important, maybe consider Optane.
 

Cutha

Member
Sep 24, 2016
75
7
8
50
Canada
The Optane solution was one that I had not thought of and I was getting pretty excited about it. Unfortunately the write speeds are < the SSD's. For sequential 128k the SM961 at QD2 is r/w 2576MB/1512MB vs Optane's 1413MB/218MB. The Optane 4k random IOPS are insane at QD1, 71953 vs 15513 for the SM961. According to TomsHardware anyway :)

Thanks for the suggestion, had me digging around for benchmarks and performance numbers. I actually thought Optane HAD to be used with the new Intel chipset. Shows what I know.

Looks like the 128GB SM961's are looking like the best option.