Optane Slog Passthrough (Or Alternatives)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Davewolfs

Active Member
Aug 6, 2015
339
32
28
I just picked up a p4801x and am trying to use it as a slog. Passthrough does not seem to be possible on OmniOS issue looks similar to here: Bug #11809: Support for P4510, P4600 NVMe passthrough on ESXi 6.7 - illumos gate - illumos

If I start the system, the kernel will panic right away and the device must be retired. Any thoughts on if there is a performance penalty by using regular data stores that go through ESXi? If I do use a regular datastore should I be simulating a NVMe controller?

Appreciate any input.
 

Davewolfs

Active Member
Aug 6, 2015
339
32
28
Thanks @gea

Is there any hope this will ever be fixed? The fix from VMWare is documented in the link below. I don’t believe I have any issues with Linux pass through. Even FreeBSD 12 seems to be broken.

VMware Knowledge Base
 

gea

Well-Known Member
Dec 31, 2010
3,141
1,182
113
DE
If you want an answer without waiting, ask where the driver developpers are around, Topicbox

Update :
see also the discussion at Topicbox
 
Last edited:

Davewolfs

Active Member
Aug 6, 2015
339
32
28
Here is a little bit of an interesting thing.

Did not work with the FreeBSD 12 installer. Things kept going into an infinite reboot unless I set things to safe mode OR (found out afterward) made sure the CPU was set to 1.

After reading this:

203874 – [patch] MSI/MSI-X interrupts don't work in VMware virtual machines

I set /boot/loader.conf to contain

hw.pci.honor_msi_blacklist=0

You can verify the value using sysctl. The default is 1. Doing this seems to correct things on FreeBSD.
 

Davewolfs

Active Member
Aug 6, 2015
339
32
28
@gea

Now that I have things working I was able to run some tests. Running on FreeBSD 12 since that is all I can get to work.

1. Using a true passthrough.

Code:
root@freebsd:~ # diskinfo -wS /dev/nvd0
/dev/nvd0
        512             # sectorsize
        100030242816    # mediasize in bytes (93G)
        195371568       # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        INTEL SSDPE21K100GA     # Disk descr.
        PHKE9055005T100EGN      # Disk ident.
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM

Synchronous random writes:
         0.5 kbytes:     21.1 usec/IO =     23.1 Mbytes/s
           1 kbytes:     23.9 usec/IO =     40.8 Mbytes/s
           2 kbytes:     21.7 usec/IO =     89.9 Mbytes/s
           4 kbytes:     21.3 usec/IO =    183.7 Mbytes/s
           8 kbytes:     24.8 usec/IO =    315.1 Mbytes/s
          16 kbytes:     42.8 usec/IO =    365.1 Mbytes/s
          32 kbytes:     70.4 usec/IO =    443.8 Mbytes/s
          64 kbytes:    126.2 usec/IO =    495.4 Mbytes/s
         128 kbytes:    206.7 usec/IO =    604.9 Mbytes/s
         256 kbytes:    320.8 usec/IO =    779.4 Mbytes/s
         512 kbytes:    540.1 usec/IO =    925.8 Mbytes/s
        1024 kbytes:   1010.2 usec/IO =    989.9 Mbytes/s
        2048 kbytes:   1931.6 usec/IO =   1035.4 Mbytes/s
        4096 kbytes:   3791.2 usec/IO =   1055.1 Mbytes/s
        8192 kbytes:   7468.5 usec/IO =   1071.2 Mbytes/s

root@freebsd:~ # smartctl -x /dev/nvme0
smartctl 7.0 2018-12-30 r4883 [FreeBSD 12.1-RELEASE amd64] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       INTEL SSDPE21K100GA
Serial Number:              
Firmware Version:                   E2010435
PCI Vendor/Subsystem ID:            0x8086
IEEE OUI Identifier:                0x5cd2e4
Controller ID:                      0
Number of Namespaces:               1
Namespace 1 Size/Capacity:          100,030,242,816 [100 GB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            5cd2e4 314e790100
Local Time is:                      Sun Feb 23 21:03:55 2020 UTC
Firmware Updates (0x02):            1 Slot
Optional Admin Commands (0x0007):   Security Format Frmw_DL
Optional NVM Commands (0x0006):     Wr_Unc DS_Mngmt
Maximum Data Transfer Size:         32 Pages

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +    10.00W       -        -    0  0  0  0        0       0

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         2
 1 -     512       8         2
 2 -     512      16         2
 3 -    4096       0         0
 4 -    4096       8         0
 5 -    4096      64         0
 6 -    4096     128         0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        34 Celsius
Available Spare:                    100%
Available Spare Threshold:          0%
Percentage Used:                    0%
Data Units Read:                    111,317 [56.9 GB]
Data Units Written:                 452,451 [231 GB]
Host Read Commands:                 489,863
Host Write Commands:                5,821,731
Controller Busy Time:               4
Power Cycles:                       22
Power On Hours:                     36
Unsafe Shutdowns:                   19
Media and Data Integrity Errors:    0
Error Information Log Entries:      0

Error Information (NVMe Log 0x01, max 64 entries)
No Errors Logged
2. Using an NVME Controller on a datastore. Eagerly zeroed.

Code:
root@freebsd:~ # diskinfo -wS /dev/nvd0
/dev/nvd0
        512             # sectorsize
        8589934592      # mediasize in bytes (8.0G)
        16777216        # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        Virtual disk    # Disk descr.
        VMWare NVME-0000        # Disk ident.
        No              # TRIM/UNMAP support
        0               # Rotation rate in RPM

Synchronous random writes:
         0.5 kbytes:     48.5 usec/IO =     10.1 Mbytes/s
           1 kbytes:     48.3 usec/IO =     20.2 Mbytes/s
           2 kbytes:     49.6 usec/IO =     39.4 Mbytes/s
           4 kbytes:     46.1 usec/IO =     84.7 Mbytes/s
           8 kbytes:     53.4 usec/IO =    146.4 Mbytes/s
          16 kbytes:     70.9 usec/IO =    220.4 Mbytes/s
          32 kbytes:    117.5 usec/IO =    266.0 Mbytes/s
          64 kbytes:    165.6 usec/IO =    377.5 Mbytes/s
         128 kbytes:    275.1 usec/IO =    454.4 Mbytes/s
         256 kbytes:    371.3 usec/IO =    673.3 Mbytes/s
         512 kbytes:    597.4 usec/IO =    836.9 Mbytes/s
        1024 kbytes:   1042.1 usec/IO =    959.6 Mbytes/s
        2048 kbytes:   1977.9 usec/IO =   1011.2 Mbytes/s
        4096 kbytes:   3789.7 usec/IO =   1055.5 Mbytes/s
        8192 kbytes:   7434.8 usec/IO =   1076.0 Mbytes/s

root@freebsd:~ # smartctl -x /dev/nvme0
smartctl 7.0 2018-12-30 r4883 [FreeBSD 12.1-RELEASE amd64] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       Virtual disk
Serial Number:                      VMWare NVME-0000
Firmware Version:                   1.0
PCI Vendor/Subsystem ID:            0x15ad
IEEE OUI Identifier:                0x565000
Controller ID:                      0
Number of Namespaces:               1
Namespace 1 Size/Capacity:          8,589,934,592 [8.58 GB]
Namespace 1 Formatted LBA Size:     512
Local Time is:                      Sun Feb 23 21:55:59 2020 UTC
Firmware Updates (0x03):            1 Slot, Slot 1 R/O
Maximum Data Transfer Size:         256 Pages

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     0.00W       -        -    0  0  0  0        0       0

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        11759 Celsius
Available Spare:                    1%
Available Spare Threshold:          100%
Percentage Used:                    0%
Data Units Read:                    110680464442257309696
Data Units Written:                 92233720368547758080
Host Read Commands:                 110680464442257309696000
Host Write Commands:                92233720368547758080000
Controller Busy Time:               92233720368547758080
Power Cycles:                       184467440737095516160
Power On Hours:                     1106804644422573096960
Unsafe Shutdowns:                   0
Media and Data Integrity Errors:    0
Error Information Log Entries:      0

Error Information (NVMe Log 0x01, max 4 entries)
No Errors Logged
3. Using a datastore with a SCSI controller. Eagerly zeroed.

Code:
root@freebsd:~ # diskinfo -wS /dev/da1
/dev/da1
        512             # sectorsize
        8589934592      # mediasize in bytes (8.0G)
        16777216        # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        1044            # Cylinders according to firmware.
        255             # Heads according to firmware.
        63              # Sectors according to firmware.
        VMware Virtual disk     # Disk descr.
                        # Disk ident.
        No              # TRIM/UNMAP support
        0               # Rotation rate in RPM
        Not_Zoned       # Zone Mode

Synchronous random writes:
         0.5 kbytes:     57.8 usec/IO =      8.4 Mbytes/s
           1 kbytes:     58.1 usec/IO =     16.8 Mbytes/s
           2 kbytes:     58.7 usec/IO =     33.3 Mbytes/s
           4 kbytes:     57.0 usec/IO =     68.6 Mbytes/s
           8 kbytes:     62.3 usec/IO =    125.3 Mbytes/s
          16 kbytes:     74.6 usec/IO =    209.4 Mbytes/s
          32 kbytes:     91.3 usec/IO =    342.3 Mbytes/s
          64 kbytes:    155.3 usec/IO =    402.5 Mbytes/s
         128 kbytes:    254.1 usec/IO =    492.0 Mbytes/s
         256 kbytes:    373.1 usec/IO =    670.1 Mbytes/s
         512 kbytes:    596.8 usec/IO =    837.8 Mbytes/s
        1024 kbytes:   1058.5 usec/IO =    944.7 Mbytes/s
        2048 kbytes:   1974.2 usec/IO =   1013.1 Mbytes/s
        4096 kbytes:   3810.5 usec/IO =   1049.7 Mbytes/s
        8192 kbytes:   7463.4 usec/IO =   1071.9 Mbytes/s

root@freebsd:~ # smartctl -x /dev/da1
smartctl 7.0 2018-12-30 r4883 [FreeBSD 12.1-RELEASE amd64] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:               VMware
Product:              Virtual disk
Revision:             2.0
Compliance:           SPC-4
User Capacity:        8,589,934,592 bytes [8.58 GB]
Logical block size:   512 bytes
LU is fully provisioned
Rotation Rate:        Solid State Device
Device type:          disk
Local Time is:        Sun Feb 23 22:02:13 2020 UTC
SMART support is:     Unavailable - device lacks SMART capability.
Read Cache is:        Unavailable
Writeback Cache is:   Unavailable

=== START OF READ SMART DATA SECTION ===
Current Drive Temperature:     0 C
Drive Trip Temperature:        0 C

Error Counter logging not supported

Device does not support Self Test logging
Device does not support Background scan results logging
Passthrough > NVME Controller > SCSI Controller

Seems that impact is greatest for small block sizes. Hardware tested on is a XeonD 1541.

Is there any tuning that can be done here?

What block size is most significant for a fast Slog?
 
Last edited:

vangoose

Active Member
May 21, 2019
326
104
43
Canada
@gea

Now that I have things working I was able to run some tests. Running on FreeBSD 12 since that is all I can get to work.

1. Using a true passthrough.

Code:
root@freebsd:~ # diskinfo -wS /dev/nvd0
/dev/nvd0
        512             # sectorsize
        100030242816    # mediasize in bytes (93G)
        195371568       # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        INTEL SSDPE21K100GA     # Disk descr.
        PHKE9055005T100EGN      # Disk ident.
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM

Synchronous random writes:
         0.5 kbytes:     21.1 usec/IO =     23.1 Mbytes/s
           1 kbytes:     23.9 usec/IO =     40.8 Mbytes/s
           2 kbytes:     21.7 usec/IO =     89.9 Mbytes/s
           4 kbytes:     21.3 usec/IO =    183.7 Mbytes/s
           8 kbytes:     24.8 usec/IO =    315.1 Mbytes/s
          16 kbytes:     42.8 usec/IO =    365.1 Mbytes/s
          32 kbytes:     70.4 usec/IO =    443.8 Mbytes/s
          64 kbytes:    126.2 usec/IO =    495.4 Mbytes/s
         128 kbytes:    206.7 usec/IO =    604.9 Mbytes/s
         256 kbytes:    320.8 usec/IO =    779.4 Mbytes/s
         512 kbytes:    540.1 usec/IO =    925.8 Mbytes/s
        1024 kbytes:   1010.2 usec/IO =    989.9 Mbytes/s
        2048 kbytes:   1931.6 usec/IO =   1035.4 Mbytes/s
        4096 kbytes:   3791.2 usec/IO =   1055.1 Mbytes/s
        8192 kbytes:   7468.5 usec/IO =   1071.2 Mbytes/s

root@freebsd:~ # smartctl -x /dev/nvme0
smartctl 7.0 2018-12-30 r4883 [FreeBSD 12.1-RELEASE amd64] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       INTEL SSDPE21K100GA
Serial Number:             
Firmware Version:                   E2010435
PCI Vendor/Subsystem ID:            0x8086
IEEE OUI Identifier:                0x5cd2e4
Controller ID:                      0
Number of Namespaces:               1
Namespace 1 Size/Capacity:          100,030,242,816 [100 GB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            5cd2e4 314e790100
Local Time is:                      Sun Feb 23 21:03:55 2020 UTC
Firmware Updates (0x02):            1 Slot
Optional Admin Commands (0x0007):   Security Format Frmw_DL
Optional NVM Commands (0x0006):     Wr_Unc DS_Mngmt
Maximum Data Transfer Size:         32 Pages

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +    10.00W       -        -    0  0  0  0        0       0

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         2
 1 -     512       8         2
 2 -     512      16         2
 3 -    4096       0         0
 4 -    4096       8         0
 5 -    4096      64         0
 6 -    4096     128         0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        34 Celsius
Available Spare:                    100%
Available Spare Threshold:          0%
Percentage Used:                    0%
Data Units Read:                    111,317 [56.9 GB]
Data Units Written:                 452,451 [231 GB]
Host Read Commands:                 489,863
Host Write Commands:                5,821,731
Controller Busy Time:               4
Power Cycles:                       22
Power On Hours:                     36
Unsafe Shutdowns:                   19
Media and Data Integrity Errors:    0
Error Information Log Entries:      0

Error Information (NVMe Log 0x01, max 64 entries)
No Errors Logged
2. Using an NVME Controller on a datastore. Eagerly zeroed.

Code:
root@freebsd:~ # diskinfo -wS /dev/nvd0
/dev/nvd0
        512             # sectorsize
        8589934592      # mediasize in bytes (8.0G)
        16777216        # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        Virtual disk    # Disk descr.
        VMWare NVME-0000        # Disk ident.
        No              # TRIM/UNMAP support
        0               # Rotation rate in RPM

Synchronous random writes:
         0.5 kbytes:     48.5 usec/IO =     10.1 Mbytes/s
           1 kbytes:     48.3 usec/IO =     20.2 Mbytes/s
           2 kbytes:     49.6 usec/IO =     39.4 Mbytes/s
           4 kbytes:     46.1 usec/IO =     84.7 Mbytes/s
           8 kbytes:     53.4 usec/IO =    146.4 Mbytes/s
          16 kbytes:     70.9 usec/IO =    220.4 Mbytes/s
          32 kbytes:    117.5 usec/IO =    266.0 Mbytes/s
          64 kbytes:    165.6 usec/IO =    377.5 Mbytes/s
         128 kbytes:    275.1 usec/IO =    454.4 Mbytes/s
         256 kbytes:    371.3 usec/IO =    673.3 Mbytes/s
         512 kbytes:    597.4 usec/IO =    836.9 Mbytes/s
        1024 kbytes:   1042.1 usec/IO =    959.6 Mbytes/s
        2048 kbytes:   1977.9 usec/IO =   1011.2 Mbytes/s
        4096 kbytes:   3789.7 usec/IO =   1055.5 Mbytes/s
        8192 kbytes:   7434.8 usec/IO =   1076.0 Mbytes/s

root@freebsd:~ # smartctl -x /dev/nvme0
smartctl 7.0 2018-12-30 r4883 [FreeBSD 12.1-RELEASE amd64] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       Virtual disk
Serial Number:                      VMWare NVME-0000
Firmware Version:                   1.0
PCI Vendor/Subsystem ID:            0x15ad
IEEE OUI Identifier:                0x565000
Controller ID:                      0
Number of Namespaces:               1
Namespace 1 Size/Capacity:          8,589,934,592 [8.58 GB]
Namespace 1 Formatted LBA Size:     512
Local Time is:                      Sun Feb 23 21:55:59 2020 UTC
Firmware Updates (0x03):            1 Slot, Slot 1 R/O
Maximum Data Transfer Size:         256 Pages

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     0.00W       -        -    0  0  0  0        0       0

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        11759 Celsius
Available Spare:                    1%
Available Spare Threshold:          100%
Percentage Used:                    0%
Data Units Read:                    110680464442257309696
Data Units Written:                 92233720368547758080
Host Read Commands:                 110680464442257309696000
Host Write Commands:                92233720368547758080000
Controller Busy Time:               92233720368547758080
Power Cycles:                       184467440737095516160
Power On Hours:                     1106804644422573096960
Unsafe Shutdowns:                   0
Media and Data Integrity Errors:    0
Error Information Log Entries:      0

Error Information (NVMe Log 0x01, max 4 entries)
No Errors Logged
3. Using a datastore with a SCSI controller. Eagerly zeroed.

Code:
root@freebsd:~ # diskinfo -wS /dev/da1
/dev/da1
        512             # sectorsize
        8589934592      # mediasize in bytes (8.0G)
        16777216        # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        1044            # Cylinders according to firmware.
        255             # Heads according to firmware.
        63              # Sectors according to firmware.
        VMware Virtual disk     # Disk descr.
                        # Disk ident.
        No              # TRIM/UNMAP support
        0               # Rotation rate in RPM
        Not_Zoned       # Zone Mode

Synchronous random writes:
         0.5 kbytes:     57.8 usec/IO =      8.4 Mbytes/s
           1 kbytes:     58.1 usec/IO =     16.8 Mbytes/s
           2 kbytes:     58.7 usec/IO =     33.3 Mbytes/s
           4 kbytes:     57.0 usec/IO =     68.6 Mbytes/s
           8 kbytes:     62.3 usec/IO =    125.3 Mbytes/s
          16 kbytes:     74.6 usec/IO =    209.4 Mbytes/s
          32 kbytes:     91.3 usec/IO =    342.3 Mbytes/s
          64 kbytes:    155.3 usec/IO =    402.5 Mbytes/s
         128 kbytes:    254.1 usec/IO =    492.0 Mbytes/s
         256 kbytes:    373.1 usec/IO =    670.1 Mbytes/s
         512 kbytes:    596.8 usec/IO =    837.8 Mbytes/s
        1024 kbytes:   1058.5 usec/IO =    944.7 Mbytes/s
        2048 kbytes:   1974.2 usec/IO =   1013.1 Mbytes/s
        4096 kbytes:   3810.5 usec/IO =   1049.7 Mbytes/s
        8192 kbytes:   7463.4 usec/IO =   1071.9 Mbytes/s

root@freebsd:~ # smartctl -x /dev/da1
smartctl 7.0 2018-12-30 r4883 [FreeBSD 12.1-RELEASE amd64] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:               VMware
Product:              Virtual disk
Revision:             2.0
Compliance:           SPC-4
User Capacity:        8,589,934,592 bytes [8.58 GB]
Logical block size:   512 bytes
LU is fully provisioned
Rotation Rate:        Solid State Device
Device type:          disk
Local Time is:        Sun Feb 23 22:02:13 2020 UTC
SMART support is:     Unavailable - device lacks SMART capability.
Read Cache is:        Unavailable
Writeback Cache is:   Unavailable

=== START OF READ SMART DATA SECTION ===
Current Drive Temperature:     0 C
Drive Trip Temperature:        0 C

Error Counter logging not supported

Device does not support Self Test logging
Device does not support Background scan results logging
Passthrough > NVME Controller > SCSI Controller

Seems that impact is greatest for small block sizes. Hardware tested on is a XeonD 1541.

Is there any tuning that can be done here?

What block size is most significant for a fast Slog?
Disable trim.

I have FreeNAS on ESXi 6.7U3 with passthrough Intel x550 VF, LSI 3008.
OS disk is vmdk using lsi sas virtual controller from datastore over nvme, keeps getting trim error.
Using nvme virtual controller give other type of error like nvme0 controller reset.
RDM a nvme namespace with virtual nvme controller give the same nvme0 controller reset error.

look at this
No # TRIM/UNMAP support

Spent hours last night to dig the issue, and came to a solution to clear all those errors. The vmware virtualized sas and nvme controller do not support trim.
vfs.zfs.trim.enabled=0

Haven't got time to test performance yet.
 

Davewolfs

Active Member
Aug 6, 2015
339
32
28
Interesting that you mention that.

I tested with Freenas 11.3 too using the setting I mentioned above. When running diskinfo I continually got an error on 256k size. Had no idea why. It’s probably related to Trim.

12.x doesn’t have this problem. OmniOS dev is so quiet there doesn’t seem to be a single response.

May I ask why you are using data stores instead of passthrough?
 

vangoose

Active Member
May 21, 2019
326
104
43
Canada
Interesting that you mention that.

I tested with Freenas 11.3 too using the setting I mentioned above. When running diskinfo I continually got an error on 256k size. Had no idea why. It’s probably related to Trim.

12.x doesn’t have this problem. OmniOS dev is so quiet there doesn’t seem to be a single response.
i have the same error on freenas 11.3 baremetal. Trim is fully supported on my nvme. There seems to be bugs in nvme on FreeBSD.

interesting thing is it failed with same error on the 24gb namespace but successful on the 1.8tb namespace when rdmed to the vm.
 

vangoose

Active Member
May 21, 2019
326
104
43
Canada
Have you run tunefs -t /dev/thedevice

You want to make sure you have

hw.pci.honor_msi_blacklist=0 in your loader.conf.
The same error I mean is 256K AIO not supported.

hw.pci.honor_msi_blacklist=0
Yes, that's the only way I could get my Intel VF passed to the freenas vm.

I'm not passthrough the entire nvme because I want to use it for different storage servers with different OS.
https://forums.servethehome.com/ind...change-to-zfsonlinux.27119/page-2#post-254975
 

Davewolfs

Active Member
Aug 6, 2015
339
32
28
Makes sense. That is good to know about

vfs.zfs.trim.enabled=0

I should try that on 11.3 to see if I can pass the diskinfo test.
 

vangoose

Active Member
May 21, 2019
326
104
43
Canada
If you check freenas forum, people have problem with ssd behind lsi sas3008.
From my understanding, many comsumer grade SSDs don't truly support trim and it's documented by LSI.
What freenas should do is to test if the disk supports trim/unmap and disable zpool trim if it fails the test.

That being said, I'm not sure if VMFS reclaim will still work once zfs trim is disabled. That's why I'm building multiple storage servers, one with all spindle disks for CIFS/NFS first.
 

Davewolfs

Active Member
Aug 6, 2015
339
32
28
@vangoose The SSD I am using is not consumer grade. So definitely a bad driver but seems to be resolved with BSD 12.

I haven't had any issues with OmniOS and the VMWare NVMe controller. It's too bad the project is running on skeleton support.