Napp-It AIO Performance Troubleshooting

[coolrunnings82]

Environment:
HP Proliant ML110 G7
Xeon E3-1220
16gb DDR3 ECC RAM
4x 500gb Western Digital RE4 (2x ZFS mirrored sets)
1x 500gb Western Digital RE4
IBM M1015 flashed to IT mode
ESXi 5.1 Update 2
OpenIndiana 151-A8 - Allocated 8gb RAM
Windows 7 Pro x64 VM's x 2 - Allocated 4gb RAM each

Problem:
1. Super slow disk performance on a relatively intermittent basis.
2. Gnome interface showing up with no menu bar at the top nor anything at the bottom - just a desktop with a couple folders.

What I've Tried/Checked:

• Checked SMART data on drives. All check out with nothing out of the ordinary and no reallocated or pending bad sectors
• Ran a scrub on the pool. Performance starts out at 2mb/s and caps out around 55mb/s. Normal?
• Ensured that NFS Sync was disabled for the VM pool.

What I need:

• Suggestions for troubleshooting the slowdown.
• Recommendations of tools/procedure that can individually test the drives to see if one is actually experiencing issues that are slowing down the overall performance.
• Suggestions for fixing Gnome interface.

Thanks for any help!!!

 

[mrkrad]

hard drives without a BBWC/FBWC is going to be painfully slow!

Also ensure your host is set to max power/disable C1e/disable C-states, and your guest vm's likewise are set to maximum power mode (no power savings!) it makes a huge difference even to this day.

And disable IPV6 on esxi 5.1, and disable flow control.

I'm not sure what nic you are using but that might be a problem - ssh into the esxi server and check /var/log/vmkernel.log and /var/log/vmkwarning.log or just do a dmesg and check for errors.

Latency warnings are a sign of death with esxi! You should not see them unless you are doing a SCSI hot-add/snapshot!

There are some tweaks you can do to increase queue depth - but my suggestion is to baseline benchmark using 1 VM on the host - then add more vm's and watch performance drop like a bandit as esxi's fair-queueing comes into play!

 

[coolrunnings82]

Thanks for the tips. I'll check tonight. Also I've not received any latency warnings from ESXi, just dealt with extreme latency causing me issues...

 

[coolrunnings82]

In the vmkwarning.log file I see lots of these:
VFAT: 4346: Failed to flush file times: Stale file handle.
LinScsi: SCSILinuxQueueCommand:1193:queuecommand failed
Pretty sure that's when I rebooted the Openindiana VM.

Not much other stuff that looks bad that I could identify... Waiting till nobody else is on then I'll change the NIC settings.