NVME Performance Under Linux

[doctor]

Does anyone have experience with various NVME drives under Linux (Centos 7 kernel 3.x, or Ubuntu w/ Kernel 4.2)? I am having rather sluggish performance on a Dell R620 and a Supermicro X9DRI board with both Samsung 960 EVO and PRO cards. The ideal configuration is MDADM raid1, but even as a single drive the performance is mediocre. The performance is generally 600-800MB/s for sequential read/writes. Doing some cursory research suggests that:

> Folks seem to commonly complain Windows operates NVME drives faster.

> Some folks are able to get solid performance out of NVME under Linux based on various hosting companies selling

Has anyone had success with specific non-super-highend disks such as S3700's?

 

[chilipepperz]

What chips do you have? Are they running PCI-E 3 not 2 speeds? V1s sometimes ran at 2 not 3.

Linux NVME is robust.

 

[EffrafaxOfWug]

First off, have you tried disabling the bitmap on the mdadm arrays? I've seen a couple of scenarios with flash arrays where the bitmap appeared to hinder performance.

What benchmark are you using that's reporting ~600MB/s reads? How are the drives mounted (as the X9DRI at least doesn't have M2 slots).

 

[vanfawx]

I have Intel P3600's installed in C6100's, C1100's and R620's, and P3700's installed in R430's. In all cases they work as expected on CentOS 7. The only non-datacenter NVMe we used was the Intel 750, which I murdered in record time due to their extremely low write endurance. But the performance while they worked was amazing

 

[Evan]

I have Intel P3600's installed in C6100's, C1100's and R620's, and P3700's installed in R430's. In all cases they work as expected on CentOS 7. The only non-datacenter NVMe we used was the Intel 750, which I murdered in record time due to their extremely low write endurance. But the performance while they worked was amazing

Out of interest what is the workload that you are having to cause that many writes ? (Just curious as everything I do is in regular enterprise and even the HPC workloads are not really heavy write, ok we do have one extreme workload that writes ~1PB as fast as the array will take it in but then there is a lot of time spent analysing the data so it’s still less than 1 dwpd

 

[vanfawx]

Graphite and MySQL both killed the Intel 750's quickly. Even the P3600's get used fairly quickly I'm finding. The downside of the Intel NVMe's is once their "life" hits 100%, they essentially die. So you get no more endurance than you pay for essentially.

 

[Evan]

Plans to check the MySQL write rates in the new year and compare to oracle now, as nearly all sql is on SAN it gets averaged out so to speak but have a few local SSD systems (none have hit 80% & 90% life though as we have warnings setup)

In general can’t disagree about the intel 750 life, and it does not officially increase with size which is odd.

New year resolution I think will be to stop even buying partial flash arrays or cached arrays but in 2018 try to go all solid state storage... let’s see but would be nice !

 

[vanfawx]

We switched to an all flash array for our shared storage about 1.5 years ago. It was a great investment and has provided consistent, predictable performance for our VMware farm.