Napp-it SuperStorage Server 6028R-E1CR12L NVMe SSD issue

[2cents]

I am running a baby sister system to Bronko's system here: Napp-it SuperStorage Server 6048R-E1CR36L Performance. with the same family of NVMe SSD.

Unfortunately, I seem to be having trouble getting the L2ARC set up on the NVMe SSD. It is installed via DirectPath I/O and it is seen by OmniOS/Napp-it, but it causes instability in the system. Not sure how to resolve.

Barebone: Supermicro | Products | SuperStorage Servers | 2U | 6048R-E1CR12L (HBA: SAS3 via LSI 3008 controller; IT mode)
CPU: 2x Intel® Xeon® Haswell-EP Series Processor E5-2630 v3, 2.40 GHz, 8-Core
RAM: 64GB (4x 16GB) Hynix DDR4-2133 CL15 RDIMM
rpool: 1x 64GB SSD SATA DOM
data-pool: 4x 6TB (P/N:0F22791 HUS726060AL5210 ) 2x2 mirror
L2ARC: 1x 400GB Intel® Solid-State Drive DC P3500 Series NVMe (Use only 300GB)
SLOG: Possibly partition 15GB off of P3500
Hypervisor: ESXi 6.0 -- 2016-03-15 (Update 2)
OS: OmniOS 5.11 omnios-r151018-ae3141d April 2016
Napp-it: 16.04f

What I know:

• I was able to attempt to partition the P3500 via Napp-it 300GB/15GB/85GB (L2ARC/SLOG/Overprovision). It choked for a long time but did come back with 296/16/88GB free.
  
  15GB SLOG not critical at this point since all sync=disable (future use)
  Overprovision due to the lower DWPD of the P3500​
  
  At this point is when I started to notice stability issues within Napp-it.

• The VMs and the file server continue to run and function without any observable issues, but anything (menus) within napp-it dealing with Disks and/or pools either takes forever to respond, or never responds (overnight).

• The OmniOS console window has a curious message shortly after boot if the P3500 is present via DirectPath. I have no physical KVM setup other than what is virtualized by ESXi?
  
• However, remote SSH via putty continues to work as expected even though the console is hung.
• I removed the DirectPath assignment and Napp-it again responds normally. I did this to load Intel's SSD Data Center Tool onto ESXi. However, the tools cannot even find the P3500 with only ESXi in the mix (No sharing with VMs). Although Intel notes that "On ESXi systems, the tool only works on Intel PCIe NVMe drives using the Intel ESXi NVMe driver.
  
  
  
  
• I then DirectPath'd the P3500 to a Win 7 VM I have set up, and it immediately recognized the P3500 and saw the partitioning done via Napp-it. I then tried the Window's version of the Data Center Tool and again it cannot see the P3500 (Possibly due to DirectPath?). This is all to try to check the firmware version, and run a self test.
• I was able to get Intel's SSD Toolbox (another tool for consumer drives) to see the P3500 and report some minimal information under Win 7. Intel says that is all this consumer tool can do, so no self test.
• I was able to unpartition the drive via Win 7 and set it back to an unprovisioned 400GB drive.

As of right now I have a nice SSD not being utilized. Does anyone know of any issues specifically related to the P3500 or my setup? Any tips on how to proceed?

​

 

[T_Minus]

Napp-IT/OmniOS so far has failed to work with any enterprise or Intel 750 consumer NVME in ESXI for me and others here.

I can pass-through the NVME to Windows, etc... no problem at all but not Napp-IT/OmniOS.

If you search my username you'll find numerous threads where people have had the same issue and I've bumped/checked-in and no updates/working from anyone yet.

 

[2cents]

Napp-IT/OmniOS so far has failed to work with any enterprise or Intel 750 consumer NVME in ESXI for me and others here.

I can pass-through the NVME to Windows, etc... no problem at all but not Napp-IT/OmniOS.

If you search my username you'll find numerous threads where people have had the same issue and I've bumped/checked-in and no updates/working from anyone yet.

 

[2cents]

T-Minus,

Thanks for the input! I referenced the Bronko thread, because he seems to have the P3700 (Same driver family) working. I also know that gea recommends the Intel P3xxx series, although it was never clear to me if it was the drive or the AOC versions.

What's your take on Bronko's setup working with the P3700 (see link at top of my post)?

Otherwise, do you have any insight or links to if/when the P3xxx series might be doable?

 

[T_Minus]

I have P3600 and P3700 and neither will work in ESXI passed-through to Napp-IT VM.

If you read my threads I mentioned GEA specifically NUMEROUS times trying to get him to tell us how to get them to work in Napp-IT VM within ESXI but so far no one has said anything.

Napp-IT / OmniOS appear to work fine with these on baremetal just not virtualized in ESXi

Bronko, from what I recall is NOT running it virtualized... he's running on baremetal.

And, like I said I can pass-through to Windows OS just fine... just not Napp-It/OmniOS

 

[2cents]

Ahh Thanks T_Minus. ESXi seems to be the culprit, although they are on the VMware approved list!

GEA.... Any thoughts or ideas?

 

[T_Minus]

Ahh Thanks T_Minus. ESXi seems to be the culprit, although they are on the VMware approved list!

GEA.... Any thoughts or ideas?

They work fine in ESXI itself.
I use 2 as VM datastores in my home AIO... but that's utilized directly by ESXi not ESXi -> OmniOS Guest.

 

[cookiesowns]

Hrm,

Just a curious thought, does your ESXi guest for OmniOS have UEFI support, and did you install OmniOS via UEFI?

 

[2cents]

cookiesowns... It is running on BIOS, but it does have UEFI support. What were your thoughts?

 

[gea]

There is a driver problem with the current Illumos nvme driver and ESXi pass-through.
There have been discussions about at illumos/omnios discuss so I would expect a driver fix in the near future.
Listbox • Email Marketing and Discussion Groups

Up to then you can only use NVMe in a barebone setup with Illumos/Omnios or via the ESXi driver

 

[2cents]

Thanks for the info./update gea.