I have a pair of Samsung SSD 960 EVO 1TB's running in software RAID1 under Centos 7 (kernel 3.10) in a Dell R620. The drives are installed in PCIe adapters, and each is on a different PCIe bus (processor 1/2).
The drives seem reasonable for reads ...
# hdparm -Tt --direct /dev/nvme0n1
/dev/nvme0n1:
Timing O_DIRECT cached reads: 4612 MB in 2.00 seconds = 2306.27 MB/sec
Timing O_DIRECT disk reads: 4098 MB in 3.00 seconds = 1365.24 MB/sec
# hdparm -Tt --direct /dev/nvme1n1
/dev/nvme1n1:
Timing O_DIRECT cached reads: 4130 MB in 2.01 seconds = 2058.76 MB/sec
Timing O_DIRECT disk reads: 3862 MB in 3.00 seconds = 1287.30 MB/sec
However write speed drops off very fast...
# dd if=/dev/zero of=test bs=1M count=200
200+0 records in
200+0 records out
209715200 bytes (210 MB) copied, 0.197257 s, 1.1 GB/s
# dd if=/dev/zero of=test bs=1M count=2000
2000+0 records in
2000+0 records out
2097152000 bytes (2.1 GB) copied, 3.75583 s, 558 MB/s
Doing a 20GB test, the server load spikes (up towards 200+) with 75% idle time, and the server log shows these:
[5145876.782688] nvme nvme1: completing aborted command with status: 0000
[5145876.793552] nvme nvme1: completing aborted command with status: 0000
[5145876.794007] nvme nvme1: completing aborted command with status: 0000
[5145876.810535] nvme nvme1: completing aborted command with status: 0000
[5145876.811053] nvme nvme1: completing aborted command with status: 0000
[5145876.811704] nvme nvme1: completing aborted command with status: 0000
[5145876.814927] nvme nvme1: I/O 800 QID 2 timeout, aborting
[5145876.814947] nvme nvme1: I/O 801 QID 2 timeout, aborting
[5145876.814951] nvme nvme1: I/O 802 QID 2 timeout, aborting
[5145876.814955] nvme nvme1: I/O 804 QID 2 timeout, aborting
[5145876.814959] nvme nvme1: I/O 805 QID 2 timeout, aborting
[5145876.814961] nvme nvme1: I/O 806 QID 2 timeout, aborting
[5145876.814964] nvme nvme1: I/O 807 QID 2 timeout, aborting
[5145876.814967] nvme nvme1: I/O 811 QID 2 timeout, aborting
# nvme smart-log /dev/nvme1
Smart Log for NVME device:nvme1 namespace-id:ffffffff
critical_warning : 0
temperature : 51 C
available_spare : 100%
available_spare_threshold : 10%
percentage_used : 3%
data_units_read : 18,907,979
data_units_written : 14,754,311
host_read_commands : 139,277,178
host_write_commands : 386,999,247
controller_busy_time : 7,947
power_cycles : 16
power_on_hours : 1,409
unsafe_shutdowns : 11
media_errors : 0
num_err_log_entries : 0
Warning Temperature Time : 0
Critical Composite Temperature Time : 0
Temperature Sensor 1 : 51 C
Temperature Sensor 2 : 77 C
The drives seem reasonable for reads ...
# hdparm -Tt --direct /dev/nvme0n1
/dev/nvme0n1:
Timing O_DIRECT cached reads: 4612 MB in 2.00 seconds = 2306.27 MB/sec
Timing O_DIRECT disk reads: 4098 MB in 3.00 seconds = 1365.24 MB/sec
# hdparm -Tt --direct /dev/nvme1n1
/dev/nvme1n1:
Timing O_DIRECT cached reads: 4130 MB in 2.01 seconds = 2058.76 MB/sec
Timing O_DIRECT disk reads: 3862 MB in 3.00 seconds = 1287.30 MB/sec
However write speed drops off very fast...
# dd if=/dev/zero of=test bs=1M count=200
200+0 records in
200+0 records out
209715200 bytes (210 MB) copied, 0.197257 s, 1.1 GB/s
# dd if=/dev/zero of=test bs=1M count=2000
2000+0 records in
2000+0 records out
2097152000 bytes (2.1 GB) copied, 3.75583 s, 558 MB/s
Doing a 20GB test, the server load spikes (up towards 200+) with 75% idle time, and the server log shows these:
[5145876.782688] nvme nvme1: completing aborted command with status: 0000
[5145876.793552] nvme nvme1: completing aborted command with status: 0000
[5145876.794007] nvme nvme1: completing aborted command with status: 0000
[5145876.810535] nvme nvme1: completing aborted command with status: 0000
[5145876.811053] nvme nvme1: completing aborted command with status: 0000
[5145876.811704] nvme nvme1: completing aborted command with status: 0000
[5145876.814927] nvme nvme1: I/O 800 QID 2 timeout, aborting
[5145876.814947] nvme nvme1: I/O 801 QID 2 timeout, aborting
[5145876.814951] nvme nvme1: I/O 802 QID 2 timeout, aborting
[5145876.814955] nvme nvme1: I/O 804 QID 2 timeout, aborting
[5145876.814959] nvme nvme1: I/O 805 QID 2 timeout, aborting
[5145876.814961] nvme nvme1: I/O 806 QID 2 timeout, aborting
[5145876.814964] nvme nvme1: I/O 807 QID 2 timeout, aborting
[5145876.814967] nvme nvme1: I/O 811 QID 2 timeout, aborting
# nvme smart-log /dev/nvme1
Smart Log for NVME device:nvme1 namespace-id:ffffffff
critical_warning : 0
temperature : 51 C
available_spare : 100%
available_spare_threshold : 10%
percentage_used : 3%
data_units_read : 18,907,979
data_units_written : 14,754,311
host_read_commands : 139,277,178
host_write_commands : 386,999,247
controller_busy_time : 7,947
power_cycles : 16
power_on_hours : 1,409
unsafe_shutdowns : 11
media_errors : 0
num_err_log_entries : 0
Warning Temperature Time : 0
Critical Composite Temperature Time : 0
Temperature Sensor 1 : 51 C
Temperature Sensor 2 : 77 C