Napp-It AIO Performance Troubleshooting

coolrunnings82

Active Member
Mar 26, 2012
399
88
28
Environment:
HP Proliant ML110 G7
Xeon E3-1220
16gb DDR3 ECC RAM
4x 500gb Western Digital RE4 (2x ZFS mirrored sets)
1x 500gb Western Digital RE4
IBM M1015 flashed to IT mode
ESXi 5.1 Update 2
OpenIndiana 151-A8 - Allocated 8gb RAM
Windows 7 Pro x64 VM's x 2 - Allocated 4gb RAM each

Problem:
1. Super slow disk performance on a relatively intermittent basis.
2. Gnome interface showing up with no menu bar at the top nor anything at the bottom - just a desktop with a couple folders.

What I've Tried/Checked:
  • Checked SMART data on drives. All check out with nothing out of the ordinary and no reallocated or pending bad sectors
  • Ran a scrub on the pool. Performance starts out at 2mb/s and caps out around 55mb/s. Normal?
  • Ensured that NFS Sync was disabled for the VM pool.

What I need:
  • Suggestions for troubleshooting the slowdown.
  • Recommendations of tools/procedure that can individually test the drives to see if one is actually experiencing issues that are slowing down the overall performance.
  • Suggestions for fixing Gnome interface.

Thanks for any help!!!
 

mrkrad

Well-Known Member
Oct 13, 2012
1,244
52
48
hard drives without a BBWC/FBWC is going to be painfully slow!

Also ensure your host is set to max power/disable C1e/disable C-states, and your guest vm's likewise are set to maximum power mode (no power savings!) it makes a huge difference even to this day.

And disable IPV6 on esxi 5.1, and disable flow control.

I'm not sure what nic you are using but that might be a problem - ssh into the esxi server and check /var/log/vmkernel.log and /var/log/vmkwarning.log or just do a dmesg and check for errors.

Latency warnings are a sign of death with esxi! You should not see them unless you are doing a SCSI hot-add/snapshot!

There are some tweaks you can do to increase queue depth - but my suggestion is to baseline benchmark using 1 VM on the host - then add more vm's and watch performance drop like a bandit as esxi's fair-queueing comes into play!
 

coolrunnings82

Active Member
Mar 26, 2012
399
88
28
Thanks for the tips. I'll check tonight. Also I've not received any latency warnings from ESXi, just dealt with extreme latency causing me issues...
 

coolrunnings82

Active Member
Mar 26, 2012
399
88
28
In the vmkwarning.log file I see lots of these:
VFAT: 4346: Failed to flush file times: Stale file handle.
LinScsi: SCSILinuxQueueCommand:1193:queuecommand failed
Pretty sure that's when I rebooted the Openindiana VM.

Not much other stuff that looks bad that I could identify... Waiting till nobody else is on then I'll change the NIC settings.