Intel SSD 750 ESXi Passthrough?

[mattlach]

Hey all,

So I have a SSD750 in my workstation, and I have come to really appreciate the drive. At about the same time that I noticed that I need a new SSD for my L2ARC cache drive on my FreeNAS ZFS guest, I also happened to notice that 400GB Intel 750 Drives can now be had for about $320!

I am considering picking one up as my new L2ARC drive. I have confirmed that FreeNAS now supports NVMe, so this shouldn't be a problem, but I am wondering how this drive might behave if installed in my ESXi hypervisor and use PCI passthrough to present it to my FreeNAS guest.

Does anyone know if the Intel 750 plays nice with PCI Passthrough?

I have reason to be concerned because other companies (*cough* Nvidia *cough*) sabotage their consumer level devices PCI passthrough support so they don't compete with their professional products, and Intel is in the same position in this market with the 750 (consumer) model and their p3500/p3600/p3700/whatever which are very similar, but targeted towards the enterprise market.

I'd appreciate any thoughts!

--Matt

 

[gea]

When I tried this lately (ESXi 6.00U1, OmniOS 151017) I was able to see the P750 in format but it hangs on a zpool import.

 

[mattlach]

When I tried this lately (ESXi 6.00U1, OmniOS 151017) I was able to see the P750 in format but it hangs on a zpool import.

Appreciate your input!

When you did this, how were you trying to use the SSD 750? As L2ARC?

 

[T_Minus]

I've only done this with the Intel Enterprise drives but was hoping to do it with the 750s for home, guess I'll have to test before planning out the entire home AIO around these for VMs :/ I'll post back when I test, I look forward to other sinput as well.

 

[mattlach]

I've only done this with the Intel Enterprise drives but was hoping to do it with the 750s for home, guess I'll have to test before planning out the entire home AIO around these for VMs :/ I'll post back when I test, I look forward to other sinput as well.

I look forward to hear what you find!

Right now I am budget constrained to the point where I can't just buy one to test if it winds up not working.

The data on the drive I have in my desktop is also a little bit too sensitive to stick in the ole ESXi server for a test.

 

[Wondersausage]

Watch the endurance on the 750 series, which is a tiny 127 TBW (compare with 4380 TBW for a DC P3600).

 

[gea]

Appreciate your input!

When you did this, how were you trying to use the SSD 750? As L2ARC?

Datapool, L2Arc, ZIL - does not matter -
I want NVMe everywhere

 

[mattlach]

Watch the endurance on the 750 series, which is a tiny 127 TBW (compare with 4380 TBW for a DC P3600).

True, but in a home ZFS server it will still last 5+ years, much longer than you'll likely want it before upgrading

The DC P3600 at the same size costs about double. Doesn't seem worth it.

 

[gea]

update

I have tried a DC P3600.
Similar result, device is seen but does not work as a pass-through device, hangs on a format.

(SuperMicro X9SRH-7TF, ESXi 6.00u1, OmniOS 151017 Jan 2016)

As ESXi supports NVMe disks, I tried a virtual disk as a disk for VMs
but results were quite disappointing with a sequential AJA test
(about 400MB/s read/write with 4k and 16bit RGB compared to the 800MB/s that is should have)

 

[T_Minus]

update

I have tried a DC P3600.
Similar result, device is seen but does not work as a pass-through device, hangs on a format.

(SuperMicro X9SRH-7TF, ESXi 6.00u1, OmniOS 151017 Jan 2016)

As ESXi supports NVMe disks, I tried a virtual disk as a disk for VMs
but results were quite disappointing with a sequential AJA test
(about 400MB/s read/write with 4k and 16bit RGB compared to the 800MB/s that is should have)

I'm not too sure what that issue is, X9 maybe?

I've passed through P3600, P3605, P3700 and shared the benchmarks here probably pushing 8 months or more ago now... (I passed through to Windows for testing... I'll try other OS soon, and 750s)

The performance wasn't the same as bare-metal if I Recall, but was def not only 400MB/s.

 

[mattlach]

update

I have tried a DC P3600.
Similar result, device is seen but does not work as a pass-through device, hangs on a format.

Hmm. Maybe try a block device passthrough, instead of a PCI Passthrough?

This guide should help.

As ESXi supports NVMe disks, I tried a virtual disk as a disk for VMs
but results were quite disappointing with a sequential AJA test
(about 400MB/s read/write with 4k and 16bit RGB compared to the 800MB/s that is should have)

I remember reading somewhere that the built in NVMe support in ESXi can be a little slow, but that Intel has a vib driver you can install that speeds things up.

Ahh, here it is

If you do the block device passthrough (or Raw Device Mapping, RDM as they call it) it will (unlike PCI passthrough) still rely on the host for drivers, so you'll still want to install Intels vib for best performance.

Good luck!

 

[T_Minus]

I just attempted to pass through the Intel 750 into napp-it so far no go.

I've successfully passed-through the enterprise intel nvme in windows though w/out issue... I haven't tried the 750 yet just napp-it so far.

Did you ever get this working @gea ?

 

[T_Minus]

@mattlach tagging you here -- did you get the passthrough working for the 750?

 

[T_Minus]

Anyone? LOL!@!

 

[ItsValium]

Regarding the pass through of nvme to a freenas guest and initialising a pool. You have to take into account the setting of zfs.vdev.trim_on_init By default when adding the drive it will trim the entire thing and that makes it lock up for a long time. If you set the tunable to disabled it will almost instantly be added and functional. If you keep that in mind and trim the drive before adding it using another system then you should be good to go.
You might give that a try!

 

[T_Minus]

I haven't tried FreeNAS as I'm using OmniOS on my AIO...

On OmniOS the drive does not show up at all. It shows up and works fine in ESXI though.