@gea
Now that I have things working I was able to run some tests. Running on FreeBSD 12 since that is all I can get to work.
1. Using a true passthrough.
Code:
root@freebsd:~ # diskinfo -wS /dev/nvd0
/dev/nvd0
512 # sectorsize
100030242816 # mediasize in bytes (93G)
195371568 # mediasize in sectors
0 # stripesize
0 # stripeoffset
INTEL SSDPE21K100GA # Disk descr.
PHKE9055005T100EGN # Disk ident.
Yes # TRIM/UNMAP support
0 # Rotation rate in RPM
Synchronous random writes:
0.5 kbytes: 21.1 usec/IO = 23.1 Mbytes/s
1 kbytes: 23.9 usec/IO = 40.8 Mbytes/s
2 kbytes: 21.7 usec/IO = 89.9 Mbytes/s
4 kbytes: 21.3 usec/IO = 183.7 Mbytes/s
8 kbytes: 24.8 usec/IO = 315.1 Mbytes/s
16 kbytes: 42.8 usec/IO = 365.1 Mbytes/s
32 kbytes: 70.4 usec/IO = 443.8 Mbytes/s
64 kbytes: 126.2 usec/IO = 495.4 Mbytes/s
128 kbytes: 206.7 usec/IO = 604.9 Mbytes/s
256 kbytes: 320.8 usec/IO = 779.4 Mbytes/s
512 kbytes: 540.1 usec/IO = 925.8 Mbytes/s
1024 kbytes: 1010.2 usec/IO = 989.9 Mbytes/s
2048 kbytes: 1931.6 usec/IO = 1035.4 Mbytes/s
4096 kbytes: 3791.2 usec/IO = 1055.1 Mbytes/s
8192 kbytes: 7468.5 usec/IO = 1071.2 Mbytes/s
root@freebsd:~ # smartctl -x /dev/nvme0
smartctl 7.0 2018-12-30 r4883 [FreeBSD 12.1-RELEASE amd64] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Number: INTEL SSDPE21K100GA
Serial Number:
Firmware Version: E2010435
PCI Vendor/Subsystem ID: 0x8086
IEEE OUI Identifier: 0x5cd2e4
Controller ID: 0
Number of Namespaces: 1
Namespace 1 Size/Capacity: 100,030,242,816 [100 GB]
Namespace 1 Formatted LBA Size: 512
Namespace 1 IEEE EUI-64: 5cd2e4 314e790100
Local Time is: Sun Feb 23 21:03:55 2020 UTC
Firmware Updates (0x02): 1 Slot
Optional Admin Commands (0x0007): Security Format Frmw_DL
Optional NVM Commands (0x0006): Wr_Unc DS_Mngmt
Maximum Data Transfer Size: 32 Pages
Supported Power States
St Op Max Active Idle RL RT WL WT Ent_Lat Ex_Lat
0 + 10.00W - - 0 0 0 0 0 0
Supported LBA Sizes (NSID 0x1)
Id Fmt Data Metadt Rel_Perf
0 + 512 0 2
1 - 512 8 2
2 - 512 16 2
3 - 4096 0 0
4 - 4096 8 0
5 - 4096 64 0
6 - 4096 128 0
=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SMART/Health Information (NVMe Log 0x02)
Critical Warning: 0x00
Temperature: 34 Celsius
Available Spare: 100%
Available Spare Threshold: 0%
Percentage Used: 0%
Data Units Read: 111,317 [56.9 GB]
Data Units Written: 452,451 [231 GB]
Host Read Commands: 489,863
Host Write Commands: 5,821,731
Controller Busy Time: 4
Power Cycles: 22
Power On Hours: 36
Unsafe Shutdowns: 19
Media and Data Integrity Errors: 0
Error Information Log Entries: 0
Error Information (NVMe Log 0x01, max 64 entries)
No Errors Logged
2. Using an NVME Controller on a datastore. Eagerly zeroed.
Code:
root@freebsd:~ # diskinfo -wS /dev/nvd0
/dev/nvd0
512 # sectorsize
8589934592 # mediasize in bytes (8.0G)
16777216 # mediasize in sectors
0 # stripesize
0 # stripeoffset
Virtual disk # Disk descr.
VMWare NVME-0000 # Disk ident.
No # TRIM/UNMAP support
0 # Rotation rate in RPM
Synchronous random writes:
0.5 kbytes: 48.5 usec/IO = 10.1 Mbytes/s
1 kbytes: 48.3 usec/IO = 20.2 Mbytes/s
2 kbytes: 49.6 usec/IO = 39.4 Mbytes/s
4 kbytes: 46.1 usec/IO = 84.7 Mbytes/s
8 kbytes: 53.4 usec/IO = 146.4 Mbytes/s
16 kbytes: 70.9 usec/IO = 220.4 Mbytes/s
32 kbytes: 117.5 usec/IO = 266.0 Mbytes/s
64 kbytes: 165.6 usec/IO = 377.5 Mbytes/s
128 kbytes: 275.1 usec/IO = 454.4 Mbytes/s
256 kbytes: 371.3 usec/IO = 673.3 Mbytes/s
512 kbytes: 597.4 usec/IO = 836.9 Mbytes/s
1024 kbytes: 1042.1 usec/IO = 959.6 Mbytes/s
2048 kbytes: 1977.9 usec/IO = 1011.2 Mbytes/s
4096 kbytes: 3789.7 usec/IO = 1055.5 Mbytes/s
8192 kbytes: 7434.8 usec/IO = 1076.0 Mbytes/s
root@freebsd:~ # smartctl -x /dev/nvme0
smartctl 7.0 2018-12-30 r4883 [FreeBSD 12.1-RELEASE amd64] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Number: Virtual disk
Serial Number: VMWare NVME-0000
Firmware Version: 1.0
PCI Vendor/Subsystem ID: 0x15ad
IEEE OUI Identifier: 0x565000
Controller ID: 0
Number of Namespaces: 1
Namespace 1 Size/Capacity: 8,589,934,592 [8.58 GB]
Namespace 1 Formatted LBA Size: 512
Local Time is: Sun Feb 23 21:55:59 2020 UTC
Firmware Updates (0x03): 1 Slot, Slot 1 R/O
Maximum Data Transfer Size: 256 Pages
Supported Power States
St Op Max Active Idle RL RT WL WT Ent_Lat Ex_Lat
0 + 0.00W - - 0 0 0 0 0 0
Supported LBA Sizes (NSID 0x1)
Id Fmt Data Metadt Rel_Perf
0 + 512 0 0
=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SMART/Health Information (NVMe Log 0x02)
Critical Warning: 0x00
Temperature: 11759 Celsius
Available Spare: 1%
Available Spare Threshold: 100%
Percentage Used: 0%
Data Units Read: 110680464442257309696
Data Units Written: 92233720368547758080
Host Read Commands: 110680464442257309696000
Host Write Commands: 92233720368547758080000
Controller Busy Time: 92233720368547758080
Power Cycles: 184467440737095516160
Power On Hours: 1106804644422573096960
Unsafe Shutdowns: 0
Media and Data Integrity Errors: 0
Error Information Log Entries: 0
Error Information (NVMe Log 0x01, max 4 entries)
No Errors Logged
3. Using a datastore with a SCSI controller. Eagerly zeroed.
Code:
root@freebsd:~ # diskinfo -wS /dev/da1
/dev/da1
512 # sectorsize
8589934592 # mediasize in bytes (8.0G)
16777216 # mediasize in sectors
0 # stripesize
0 # stripeoffset
1044 # Cylinders according to firmware.
255 # Heads according to firmware.
63 # Sectors according to firmware.
VMware Virtual disk # Disk descr.
# Disk ident.
No # TRIM/UNMAP support
0 # Rotation rate in RPM
Not_Zoned # Zone Mode
Synchronous random writes:
0.5 kbytes: 57.8 usec/IO = 8.4 Mbytes/s
1 kbytes: 58.1 usec/IO = 16.8 Mbytes/s
2 kbytes: 58.7 usec/IO = 33.3 Mbytes/s
4 kbytes: 57.0 usec/IO = 68.6 Mbytes/s
8 kbytes: 62.3 usec/IO = 125.3 Mbytes/s
16 kbytes: 74.6 usec/IO = 209.4 Mbytes/s
32 kbytes: 91.3 usec/IO = 342.3 Mbytes/s
64 kbytes: 155.3 usec/IO = 402.5 Mbytes/s
128 kbytes: 254.1 usec/IO = 492.0 Mbytes/s
256 kbytes: 373.1 usec/IO = 670.1 Mbytes/s
512 kbytes: 596.8 usec/IO = 837.8 Mbytes/s
1024 kbytes: 1058.5 usec/IO = 944.7 Mbytes/s
2048 kbytes: 1974.2 usec/IO = 1013.1 Mbytes/s
4096 kbytes: 3810.5 usec/IO = 1049.7 Mbytes/s
8192 kbytes: 7463.4 usec/IO = 1071.9 Mbytes/s
root@freebsd:~ # smartctl -x /dev/da1
smartctl 7.0 2018-12-30 r4883 [FreeBSD 12.1-RELEASE amd64] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Vendor: VMware
Product: Virtual disk
Revision: 2.0
Compliance: SPC-4
User Capacity: 8,589,934,592 bytes [8.58 GB]
Logical block size: 512 bytes
LU is fully provisioned
Rotation Rate: Solid State Device
Device type: disk
Local Time is: Sun Feb 23 22:02:13 2020 UTC
SMART support is: Unavailable - device lacks SMART capability.
Read Cache is: Unavailable
Writeback Cache is: Unavailable
=== START OF READ SMART DATA SECTION ===
Current Drive Temperature: 0 C
Drive Trip Temperature: 0 C
Error Counter logging not supported
Device does not support Self Test logging
Device does not support Background scan results logging
Passthrough > NVME Controller > SCSI Controller
Seems that impact is greatest for small block sizes. Hardware tested on is a XeonD 1541.
Is there any tuning that can be done here?
What block size is most significant for a fast Slog?