DS4243 High iowait

[lyoth]

I'm experiencing weird problem with my DS4243 where I am getting high iowait on it consistently.
Originally I had a Supermicro 846 sas2 expander back plane and load was around 8-12.
Since switching my storage over to a DS4243 I am seeing a load of 20-40.

My setup:
Supermicro 826 12 bay, supermicro x10srh-cf 128GB ram, e5-2630 v4 (running 20 instance of the same exact container that is doing the same thing, alot of small writes and reads)
DS4243 2x iom6 using sff8644 to qsfp+ cable (cable was new and purchased from 10gtek)
I tested using onboard sas3008 IT mode on my supermicro x10srh-cf (from sas card sff8643 -> sff8643 -> sff8644 -> qsfp+)
I also tested using Adaptec ASR-78165 (sff8644 -> qsfp+)
Also tested on an Supermicro sas2308 that I was previously used on my Supermicro 846 that I know is good. (from hba card sff8087 -> sff8643 -> sff8644 -> qsfp+)
All 3 different hba gave the same high load, and high iowait.
At this point, I am guessing I either have a bad sff-8644 -> qsfp+ cable, controller or the netapp backplane.
Any pointer on this would be helpful. I also have another DS4243 with iom3, and a Dell compellent controller that I have not tried yet.

My other setup running on an old Dell T320 with 10 instance of the exact same container as the other setup and I am only getting a load average of 3-4

 

[DavidWJohnston]

Is there actually a real performance problem? Is it I/O wait percentage you're looking at?

I/O Wait percentage alone may not tell you much . An older system that's more CPU-constrained could have a lower iowait for the same load, because the storage is actually less of a bottleneck, with the slower CPU taking longer to chug, so it's waiting less for the disk.

On a system with a faster CPU, the storage becomes more of a bottleneck, and the iowait% increases. This iowait percent may not be that useful - Maybe try to look at read/write latency - Or take a performance measurement from your real workload and compare them.

 

[lyoth]

Is there actually a real performance problem? Is it I/O wait percentage you're looking at?

I/O Wait percentage alone may not tell you much . An older system that's more CPU-constrained could have a lower iowait for the same load, because the storage is actually less of a bottleneck, with the slower CPU taking longer to chug, so it's waiting less for the disk.

On a system with a faster CPU, the storage becomes more of a bottleneck, and the iowait% increases. This iowait percent may not be that useful - Maybe try to look at read/write latency - Or take a performance measurement from your real workload and compare them.

That was helpful.
I eventually tracked it down to the sas3008 running too hot (80c), I attached a fan to it and temp went down to 52c and load went down about half as well