Slow NVME performance

[Nemesis_001]

Hi,

Running AIO setup, a pair of mirrored Samsung 980 1TB NVME drives. Performance seems a rather low. OmniOS r151054, ESXI 8.0U3E. Passthrough to OmniOS,
It's slower than a pool of spinners.
Any idea how come?
Thanks

start filebench..
Filebench Version 1.4.9.1
17907: 0.000: Allocated 126MB of shared memory
17907: 0.001: File-server Version 3.0 personality successfully loaded
17907: 0.002: Creating/pre-allocating files and filesets
17907: 0.009: Fileset bigfileset: 10000 files, 0 leafdirs, avg dir width = 20, avg dir depth = 3.1, 1254.784MB
17907: 0.012: Removed any existing fileset bigfileset in 1 seconds
17907: 0.012: making tree for filset /fastpool/filebench.tst/bigfileset
17907: 0.021: Creating fileset bigfileset...
17907: 0.456: Preallocated 8015 of 10000 of fileset bigfileset in 1 seconds
17907: 0.456: waiting for fileset pre-allocation to finish
17907: 0.456: Starting 1 filereader instances
17924: 0.462: Starting 50 filereaderthread threads
17907: 1.514: Running...
17907: 121.521: Run took 120 seconds...
17907: 121.523: Per-Operation Breakdown
statfile1 61717ops 514ops/s 0.0mb/s 0.0ms/op 8us/op-cpu [0ms - 33ms]
deletefile1 61717ops 514ops/s 0.0mb/s 21.1ms/op 48us/op-cpu [0ms - 96ms]
closefile3 61722ops 514ops/s 0.0mb/s 0.0ms/op 3us/op-cpu [0ms - 1ms]
readfile1 61722ops 514ops/s 67.8mb/s 0.0ms/op 25us/op-cpu [0ms - 66ms]
openfile2 61722ops 514ops/s 0.0mb/s 0.0ms/op 10us/op-cpu [0ms - 62ms]
closefile2 61722ops 514ops/s 0.0mb/s 0.0ms/op 4us/op-cpu [0ms - 4ms]
appendfilerand1 61722ops 514ops/s 4.0mb/s 22.1ms/op 29us/op-cpu [0ms - 101ms]
openfile1 61735ops 514ops/s 0.0mb/s 0.0ms/op 10us/op-cpu [0ms - 16ms]
closefile1 61735ops 514ops/s 0.0mb/s 0.0ms/op 4us/op-cpu [0ms - 3ms]
wrtfile1 61736ops 514ops/s 64.6mb/s 32.6ms/op 41us/op-cpu [0ms - 504ms]
createfile1 61756ops 515ops/s 0.0mb/s 21.1ms/op 38us/op-cpu [0ms - 96ms]
17907: 121.523:

IO Summary:
679006 ops, 5658.057 ops/s, (514/1029 r/w), 136.4mb/s, 1886us cpu/op, 32.3ms latency
17907: 121.523: Shutting down processes

ok.

Compared with a pool of 4 8TB 5400rpm spinners in RAID-Z2 configuration:

start filebench..
Filebench Version 1.4.9.1
21344: 0.000: Allocated 126MB of shared memory
21344: 0.002: File-server Version 3.0 personality successfully loaded
21344: 0.002: Creating/pre-allocating files and filesets
21344: 0.010: Fileset bigfileset: 10000 files, 0 leafdirs, avg dir width = 20, avg dir depth = 3.1, 1254.784MB
21344: 0.012: Removed any existing fileset bigfileset in 1 seconds
21344: 0.012: making tree for filset /tank/filebench.tst/bigfileset
21344: 0.022: Creating fileset bigfileset...
21344: 0.763: Preallocated 8015 of 10000 of fileset bigfileset in 1 seconds
21344: 0.763: waiting for fileset pre-allocation to finish
21344: 0.763: Starting 1 filereader instances
21361: 0.778: Starting 50 filereaderthread threads
21344: 1.781: Running...
21344: 121.830: Run took 120 seconds...
21344: 121.832: Per-Operation Breakdown
statfile1 157227ops 1310ops/s 0.0mb/s 0.0ms/op 7us/op-cpu [0ms - 23ms]
deletefile1 157216ops 1310ops/s 0.0mb/s 8.3ms/op 42us/op-cpu [0ms - 116ms]
closefile3 157238ops 1310ops/s 0.0mb/s 0.0ms/op 3us/op-cpu [0ms - 10ms]
readfile1 157238ops 1310ops/s 174.1mb/s 0.0ms/op 25us/op-cpu [0ms - 100ms]
openfile2 157238ops 1310ops/s 0.0mb/s 0.0ms/op 9us/op-cpu [0ms - 32ms]
closefile2 157238ops 1310ops/s 0.0mb/s 0.0ms/op 3us/op-cpu [0ms - 32ms]
appendfilerand1 157239ops 1310ops/s 10.2mb/s 8.6ms/op 27us/op-cpu [0ms - 116ms]
openfile1 157248ops 1310ops/s 0.0mb/s 0.0ms/op 9us/op-cpu [0ms - 74ms]
closefile1 157248ops 1310ops/s 0.0mb/s 0.0ms/op 3us/op-cpu [0ms - 15ms]
wrtfile1 157249ops 1310ops/s 164.4mb/s 12.7ms/op 40us/op-cpu [0ms - 294ms]
createfile1 157266ops 1310ops/s 0.0mb/s 8.4ms/op 33us/op-cpu [0ms - 115ms]
21344: 121.832:

IO Summary:
1729645 ops, 14407.827 ops/s, (1310/2620 r/w), 348.8mb/s, 1606us cpu/op, 12.7ms latency
21344: 121.832: Shutting down processes

ok.

 

[gea]

I would
- test the two nvme individually (in case one is bad)
- compare with a virtual disk under ESXi control (rule out passthrough problem)
- optionally: compare with a barebone OmniOS

 

[Nemesis_001]

Thanks for the replies. I need to figure out a non destructive way to test it. Have guest os running on it. Maybe move the guest to the spinners temporarily.