ESXI 6U1 passthrough with Solaris 11.3

[ehfortin]

Hi,

I've been using Solaris 11.3 with a LSI controller in a physical server for the last few weeks without any problem. Solaris recognize the LSI controller and all my NIC and I was getting very good performance result so I can see that the drivers are good.

Now, I installed ESXI 6 on the same server, did a PCI passthrough of the LSI controller and of one of my NIC and assign those to a new Solaris 11.3 VM. When I boot the CD to install Solaris, it start booting, display "Probing for device nodes..." and stop there for a while. Then, it goes with "Requesting System Maintenance Mode" followed by "Console login service(s) cannot run" and then wait for the username for system maintenance".

If I remove the LSI controller from the VM but keep the NIC, it will go through. Re-adding the LSI controller will make the VM stopping just after saying "SunOS Release 5.11 Version 11.3 64-bit".

So, I don't understand what the issue is as the same LSI card is detected and work fine with Solaris 11.3 on physical server but not when the LSI card is physical on a virtual server.

Any idea of what may be the issue? I assume it has something to do with how ESXI handle the passthrough at the VM level and that's why I've wrote this here instead of in the Solaris section.

I'll continue googling on this but if somebody has a solution or know what has to be adjusted to make it work, it would be great.

Thank you.

ehfortin

 

[whitey]

I know I've used VT-d passthru on LSi HBA's on Solaris AIO VM configs (using OmniSo curretnly in this config). Never had an issue before, make sure you have a floppy attached during install, i think I've heard if you strip it the install will fail and I know a buddy of mine was monkeying around w/ a Solaris AIO config and something silly was shooting him in the foot.

 

[ehfortin]

I know I've used VT-d passthru on LSi HBA's on Solaris AIO VM configs (using OmniSo curretnly in this config). Never had an issue before, make sure you have a floppy attached during install, i think I've heard if you strip it the install will fail and I know a buddy of mine was monkeying around w/ a Solaris AIO config and something silly was shooting him in the foot.

Floppy is still there (which is a first as I usually remove it). Will do some testing without it as well.

I just realize that ESXI is providing a virtual LSI controller to the VM as well. I'll play around this as well and report the results.

It is, for sure, something silly but... that's often the kind of issue that are the hardest to figure.

 

[ehfortin]

Well, I tried with and without floppy and got the same results. I also remove the virtual LSI parallel controller, used an IDE virtual disk instead and got the same result. That was worth the try but doesn't seems to be related to the issue.

 

[ehfortin]

As a follow-up, I found a single post saying that Solaris 11.x mpt/mpt_sas driver is not working in passthrough mode on ESXI 5/6. With version 11.1, it was said that you could replace the driver with the OmniOS version and it would work. I tried with Solaris 11.3 without success. There are errors about not being able to initialize/restart/attach the mpt device.

So, I'll do some testing passing the SSD drives through RDM to the VM and still give the NIC as a passthrough. I think most of the performance drop I saw while testing are mostly related to the virtualization of the NIC as I was able to get the speed of the SSD drive locally through a VMDK in a datastore (not even RDM).

Did somebody tested the impact of RDM compared to direct access with passthrough on fast disks?

Thanks.

 

[whitey]

Can you share a link, I haven't ran AIo on Solaris in a few years but that just sounds strange. What did Oracle break vt-D in Solaris x86? Time to consider OmniOS good sir!

 

[ehfortin]

Can you share a link, I haven't ran AIo on Solaris in a few years but that just sounds strange. What did Oracle break vt-D in Solaris x86? Time to consider OmniOS good sir!

Sure. The first link I found was this one about the LSI 2308/9207 (both number are related, 9205/9207 being the card and 2308 being the chipset if I understand correctly). I have an HP H220 which is the same card. I tried with original firmware as well as with the LSI P20.04 firmware and it has the same issue as the one described by this post:

Solaris 11.1 / ZFS with LSI 2308

Then I found another link in a Oracle forum saying that it was reported a while ago and is still a problem with Solaris 11.3 (LSI 9207 does not work as passthrough on ESXi | Oracle Community)

I tried using the OmniOS drivers but it didn't work. I'll now retry it as I upgrade the firmware to P20.04 this morning so it may be why the OmniOS drivers were not working previously. Hopefully it will work and the performance will be good. If not, I'll revert to physical and install Virtualbox on Solaris 11.3 to have a few service-VM (dhcp, dns, ad, etc).

Will let you know how it goes with the OmniOS drivers.

 

[ehfortin]

@whitey, I was looking at your signature and I saw that you have an hybrid array on OmniOS and an AFA vSAN (nice...). What kind of performance do you get from your hybrid array for multiple VM? And what you array is made of just to get an idea? Right now, I've decided to consolidate the lab VM on a single storage host as I want to be able to break apart my ESXI 6U1 cluster at will (which is not really possible if vSAN is my main storage) and as budget is limited, I'm trying to figure how I could leverage SSD. I have a few 850 EVO 250 GB but... they are not really good at sustaining any server load. I could put 4 together (2+2) to increase the performance but my feeling is that it won't help much as they are not designed for this task. Going with enterprise SSD seems like a better option but... starting from scratch, if I go with a single S3610 or SM863 (800-960 Go) will limit performance while going with multiple... it will break the bank. Buying 2x 400 Go is about the same price but will give limited capacity. Going with a single disk is safer than doing RAID0 which increase the risk by a lot. However, the upgrade path will be brutal as it will go from 1 to 4 disks (RAID10 2+2).

I guess there is not a lot of options. I could deal with 2 SSD datastore but each will be limited to the performance of a single disk. I certainly hope an enterprise SSD is a lot better then the 850 EVO because right now, I'm not impress when you create some load for more then 30 seconds or so.

Any suggestions or comments are welcomed. Now back to original programming.

 

[ehfortin]

Did try the OmniOS drivers (latest release) on Solaris 11.3 and I got error message like

"WARNING: /pci@adress (mpt0): IOC restart failed"
"failed to power up"
"mpt0: Raise power request failed"
"mpt pm initialization fialed"
"attach failed"
** repeat a few time **
then it reboot by itself. So... not working. Will switch to OmniOS for a few days and see how it goes. I was hoping to have Solaris 11.3 to work as it is supposed to include a new dedup algorithm that is supposed to be very good. If OmniOS is not conclusive, I'll revert to Solaris 11.3 on physical.

Let the adventure continue...

 

[whitey]

@whitey, I was looking at your signature and I saw that you have an hybrid array on OmniOS and an AFA vSAN (nice...). What kind of performance do you get from your hybrid array for multiple VM? And what you array is made of just to get an idea? Right now, I've decided to consolidate the lab VM on a single storage host as I want to be able to break apart my ESXI 6U1 cluster at will (which is not really possible if vSAN is my main storage) and as budget is limited, I'm trying to figure how I could leverage SSD. I have a few 850 EVO 250 GB but... they are not really good at sustaining any server load. I could put 4 together (2+2) to increase the performance but my feeling is that it won't help much as they are not designed for this task. Going with enterprise SSD seems like a better option but... starting from scratch, if I go with a single S3610 or SM863 (800-960 Go) will limit performance while going with multiple... it will break the bank. Buying 2x 400 Go is about the same price but will give limited capacity. Going with a single disk is safer than doing RAID0 which increase the risk by a lot. However, the upgrade path will be brutal as it will go from 1 to 4 disks (RAID10 2+2).

I guess there is not a lot of options. I could deal with 2 SSD datastore but each will be limited to the performance of a single disk. I certainly hope an enterprise SSD is a lot better then the 850 EVO because right now, I'm not impress when you create some load for more then 30 seconds or so.

Any suggestions or comments are welcomed. Now back to original programming.

Yep, I have long had/ran (5-6 years now) ZFS boxes, started off w/ Solaris 11 express on phys, moved to OmniOS a few years back in AIO configs, never looked back. I can serve up multi-proto (NFS/iSCSI/SMB) and get all the benefits of the true cutting edge development that Illumos community is committing to code. My ZFS hybrid array is composed of 6 1.5 tb HGST ultrastar ent magentic drives in stripped mirrors (al la raid 10) w/ a 100GB HGST hussl4010 drive for ZIL/L2ARC for each. My vSAN is composed of a s3700 200gb for cache tier and a s3610 800gb for capacity tier (x's 3 for vSAN requirement acorss 3-nodes). I got smokin perf out of either quite honestly w/ the AFA winning out probably. Put it to ya this way, i can push 600MB/sec of 'actual' sVMotion traffic in/out of each of those arrays, I think abt 10K iops on the hybrid array and nearly 20K iops outta the vSAN AFA config and it will take the beating 24/7 w/ those ent-class ssd drives, no performance drop-off 10-30 mins into workload I/O stress test.

It's costs a pretty penny but if you go used you can do it cheap, i got my 3610's for $215 each and the hussl's for $75 each. Just gotta HUNT! Raid-10 (stripped mirrored ZFS pools ALL day long for me unless it's ssd pools, I may consider raidz/2).

EDIT/DISCLOSURE: I am a AIO/stg appliance junkie, I've pretty much tried the gambit of ZFS based (OminOS, FreeNAS, ZoL), as well as btrfs (rockstor) and everything in between probably.

 

[whitey]

Did try the OmniOS drivers (latest release) on Solaris 11.3 and I got error message like

"WARNING: /pci@adress (mpt0): IOC restart failed"
"failed to power up"
"mpt0: Raise power request failed"
"mpt pm initialization fialed"
"attach failed"
** repeat a few time **
then it reboot by itself. So... not working. Will switch to OmniOS for a few days and see how it goes. I was hoping to have Solaris 11.3 to work as it is supposed to include a new dedup algorithm that is supposed to be very good. If OmniOS is not conclusive, I'll revert to Solaris 11.3 on physical.

Let the adventure continue...

I think your gonna like OmniOS once you come up to speed (I can send some good links if you need), their general administration guide is goiod to get off the ground, Gea's all-in-one pdf's, and other wiki docs off the OmniOS/Illumos groups.

I'd definitely go ent ssd's if you can stomach the cost over time. Do you have a link to Sol11.3 I could download and try, I don't have a Oracle login (dunno if they are free/easy to setup) but I'd have to go bug a colleague of mine at work to get his creds. I can test on my end if you give me the .iso.

 

[ehfortin]

I think your gonna like OmniOS once you come up to speed (I can send some good links if you need), their general administration guide is goiod to get off the ground, Gea's all-in-one pdf's, and other wiki docs off the OmniOS/Illumos groups.

I'd definitely go ent ssd's if you can stomach the cost over time. Do you have a link to Sol11.3 I could download and try, I don't have a Oracle login (dunno if they are free/easy to setup) but I'd have to go bug a colleague of mine at work to get his creds. I can test on my end if you give me the .iso.

I tried the latest release of OmniOS with my ConnectX-2 NIC (10 Gbps) and it is not supported anymore as reported in another thread (Mellanox ConnectX-2 not supported on new OmniOS HCL) so I can't think about doing physical or using passthrough in ESXI. Solaris 11.3 still work with this NIC as well as with my HBA. If I have to use virtual NIC to use OmniOS, will I loose a lot in performance and latency?

I think you are right that downloading Solaris 11.3 ask you to have an account at Oracle (OTN). However it is free to do so. Here is the webpage to download the ISO. Oracle Solaris 11.3 Downloads - Installation from CD/DVD or USB

Thanks.

 

[whitey]

Text installer seems happy w/ me so far. Those HITACHI's you see are vt-D devices hangin off a LSI 2008 HBA. Finishing up install and will lay down zpool off vt-D LSI HBA.




Something on your end for sure, works great here w/ LSI 2008 HBA and FW ver 20 IT mode on Solaris 11.3. Only reason I couldn't import that pool is cause it had OmniOS zpool version laid down (feature flag pool ver 5000). I could destroy as I have nothing on it and pound w/ some data if that makes anyone feel better.

Put the myth to bed, sorry, time to hunt the gremlins!

 

[whitey]

Couple more pics, I decided to blow away my old OmniOS zpool and lay down v37 zpool (JUST for you, dontcha feel special heh) from sol 11.3. I then setup for nfs, mounted to vSphere and initiated a sVMotion from my hybrid pool to a raidz of 4 hussl4010ass drives on the sol 11.3 AIO

 

[ehfortin]

Actually, you don't have the same LSI hba as the problematic one. The problem is with anything that is based on LSI 2308 or, at least, more specifically with LSI 9205 and LSI9207 and the equivalent from HP (H220/H221). As that's what I have, I'm stuck. Here are the combination available:

Physical with Solaris 11.3: ConnectX-2 and HP H220 work fine
Physical with OmniOS: HP220 work fine but not ConnectX-2
Virtual with passthrough on Solaris 11.3: HP H220 doesn't work at all. ConnectX-2 generate timeout so may not work.
Virtual with passthroung on OmniOS: HP H220 work fine. ConnectX-2 doesn't work

As H220 work with OmniOS with passthrough (same with Windows and Linux), I have to think the problem is at some level with Solaris 11.3. As the same card work when booting Solaris 11.3 on physical hardware, I have to assume the problem come from ESXI. Either way there is an issue with the combination of both.

So I have to choose to run on physical or on virtual with either RDM (Sol11) or virtual NIC (OmniOS). On the virtual side, which option would be the less intrusive on performance/latency? RDM or virtual NIC over 10 Gbps? Any experience with that?

 

[whitey]

Goodness gracious that is just a sas2 chip from LSI, albeit a 2300 series instead of 2200 but still that is freaking weak, get a 9211-8i, H310, H200, M1015 (all proven tanks in AIO ZFS setups) and be done w/ that nonsense...if it truly is the HBA causing so many issues why not just replace it, not too attached to it are you? Could probably unload it and get another one and maybe even come out on top if you shop smart. I know Intel X520-da1/2's are DIRT cheap now as well, that's why i stick to WELL known HW that I know if rock solid w/ these setups but honestly you didn't stray too far off the path, very odd indeed.

My vote if you don't get new devices (HBA and nic if you really need Solaris 11.3) is to go w/ last option (OmniOS AIO w/ H220) and get a new Intel X520-da1/2.

I'd bypass all that rdm business although a vnic is what you will always have on an AIO (granted it is backed by 10G at ESXi layer), never see much of a performance degradation issue there, can push damn near line rate 10G between AIO VM's and consumers of stg granted the raid set/iops are there.

 

[whitey]

I'd be willing to send ya a H310 flashed to IT mode v20 in exchange for your 9207, I think the 9207-'s were kinda highly desirable for some odd reason but apparently not here...either way offer is out there, not trying to rip you off but I'm willing to roll the dice. Also got a Intel 10G XF SR server adapter (single port) that I have no use for if that helps at all, I know that one works w/ Solaris/OmniOS/vSphere/everything. Has optic built in/on.

EDIT: Don't even waste your time passing through vt-D a 10G nic, a vswitch backed by a 10G nic in vSphere and using a vmxnet3 10G nic in the AIO will haul balls!

 

[ehfortin]

I'd be willing to send ya a H310 flashed to IT mode v20 in exchange for your 9207, I think the 9207-'s were kinda highly desirable for some odd reason but apparently not here...either way offer is out there, not trying to rip you off but I'm willing to roll the dice. Also got a Intel 10G XF SR server adapter (single port) that I have no use for if that helps at all, I know that one works w/ Solaris/OmniOS/vSphere/everything. Has optic built in/on.

EDIT: Don't even waste your time passing through vt-D a 10G nic, a vswitch backed by a 10G nic in vSphere and using a vmxnet3 10G nic in the AIO will haul balls!

As the H220 work in passthrough mode with OmniOS, I'll do more testing with that setup and a virtualized 10 Gbps NIC. The Mellanox work very well with ESXI 6U1 so... based on your comment that vNIC is pretty much as good as physical, I should have something running soon. Will let you know.

 

[ehfortin]

First step is done. OmniOS is installed, using the H220 in passthrough mode with a 10GB vNIC. First test show near 200 MB/sec in write (but unable to sustain it) and 300 MB/sec in read (pretty much sustained). I'm already using a MTU of 9000 and as I'm back to back between two ESXI host, nothing else can interfere. I'll try to reproduce some of @gea performance test. I know my SSD disk is not an enterprise model so I understand it will vary a lot between peak and sustained. I'll try to figure where is the bottleneck. I'll create a ramdisk to do some testing between a VM on ESXI01 and OmniOS on ESXI02. This should get higher performance then my current SSD.

Woohoo, finally a setup that may become my official storage server for my cluster...