Optane 900P and Openindiana 2017.10 no luck

[vinceflynow]

I tried using the Intel Optane 900P 280GB AIC with Openindiana 2017.10 and the latest nvme driver, but could not get the card detected. I also have an Intel P3700 AIC that works in the same setup.

Has anyone had luck the with the Optane 900P with either Openindiana or OmniOS?

 

[gea]

I have also ordered some Optane 900P U.2 to test them in the new X11SPH-nCTPF and X11SPH-nCTF that will arrive in mid december.

As any new technology we must see if the current driver supports them. If its more than an NVMe release problem that you can try to fix in /kernel/drv/nvme.conf where you can try to comment out any settings (+reboot),

If this does not help (or if so), please send an email to illumos discuss or -developer with a feature request to support Optane 900p at Issues - illumos.org

 

[Rand__]

Is that baremetal or in a VM?

Tried OmniOS 151024ce (VM) recently and no go. Also tried FreeBSD (Freenas) - works on baremetal but not on VM. Windows works both ways.
So Unix flavours and VMs seem to be an issue

 

[gea]

I have just got two Intel Optane 900P (U.2 with M.2 Adapter) and tried them
in a Supermicro X10SDV-4C-7TP4F mainbord and OmniOS 151024-ce and it worked (baremetal setup)

 

[Rand__]

So same pattern...
On the other hand the card is working fine in Esx when not passing through

 

[vinceflynow]

My failed test, was with pass-through PCI to Openindiana 2017.11.

 

[Kryax]

Yea there is a bug report for FreeBSD (Freenas) here talking about passthrough of the 900P to a Freenas VM:

Bug #26508: Intel Optane 900p will not work in ESX passthrough - FreeNAS - iXsystems & FreeNAS Redmine

Hopefully they are able to get the drivers sorted out sooner than later since I was looking forward to getting a 900P to use as a SLOG/L2ARC.

 

[Rand__]

Slow proceedings and I am not sure yet it's fixable by FN or Solaris* (community) devs... Baremetal works after all

 

[gea]

It depends on use case.
Each of my ESXi servers use my napp-it appliance to provide local high performance ZFS storage for local VMs with the comfort of the web appliance to increase availabilty. No single point of failure for my ESXi servers.

I prefer a high performance pools build from fast SSDs or NVMes only with build in powerloss protection. In such a use case there is no need for an slog, in contrast the slog is slower than using the onpool ZIL with the combined write performance of a pool. This means I mainly want a pool build from 2-10 Optane 900P for a usable poolsize from 280 GB to around 2TB.

This is different to a pool from disks or slower SSDs especially if they lack powerloss protection. In such a case I would use one 900P as ESXi bootdisk, for the local storage VM and virtual disks for Slog or L2Arc. As there is not trim, garbage collection or erase prior write neded and as there is no controller or disk cache involved this would be fast and secure.

 

[i386]

In such a case I would use one 900P as ESXi bootdisk, for the local storage VM and virtual disks for Slog or L2Arc.

This is the first time I read about slog devices being virtual disks., sounds like a good idea.
Have you ever compared the performance of such a design vs a baremetal/dedicated slog device?

 

[_alex]

i think only slog won't saturate an optane, and its also not very sensitive to simultanous read and write workloads. so there is quite some bandwidth left for other io besides slog, like bootdisk/l2arc or maybe also another vm running on it.
I could even imagine a single optane serving multiple pools as slog.

 

[epicurean]

hi @gea,
Could you explain in detail exactly how to setup an Optane 900P AIC as ESXI boot disk +local storage VM and virtual disk for SLOG? I understand I cannot pass through the Optane to the Napp It VM in esxi

 

[Rand__]

Its quite simple - install ESX to the optane drive, add your storage vm on the same datastore.
Then go to your vm configuration and add a second harddrive for the slog, size depending on your need, as eager zeroed and residing on the optane datastore.

Specifics will depend on the hypervisor of your choice For ESXi:

 

[gea]

Yes, currently you must create an ESXi datastore and use vdisks.

btw
I have just tested pass-through with an 900P on the new Solaris 11.4 and it works.
As there is ongoing work on the NVMe driver at Illumos (also related to the NVMe
virtual driver in ESXi) I hope this will work on Illumos as well in future.

 

[epicurean]

Sorry, I don't quite understand. @Rand__ Do you mean that instead of the usual USB drive, I install esxi on the Optane itself? and with the same Optane, also create a datastore and SLOG? How do you do that? Sorry, I am more familiar with the more "traditional" ways of installing esxi and passthrough PCI devices.

 

[Rand__]

You asked for an AIO based on Optane only, in that case yes
O/c you can also install ESXi on an USB drive, then place the storage VM on either datastore and put the secondary disk (slog) on the 900p. That's how I did it on my installations.
Just need to remember that your VM uses two datastores then if you ever have a failure or need to work on datastore level.