SAS3 HBA with SAS2 expander and SAS3 disks

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

hatchi

Member
Nov 8, 2014
59
20
8
43
Hi All,

I am facing IOPS issue with servers using SAS2 expanders vs directly connecting the disks to HBA

Let me first explain the issue

I have 2 servers supermicro 36 disks
1 of them does not have backplane and the other have 2x backplane(one in front and one on the back)
in both of them I have 8TB HGST SAS3 disks identical , same with ram and cpu

in the one with no backplane I am connecting the disks to 4x SAS3008 HBA
in the other one I have 2x sas2308 HBA (one connected to 24 disks and the other to 12)

the IOPS on the one with 4x SAS3008 is significantly higher than the with backplanes and 2x sas2308 HBA

I am not concerned about throughput but IOPS is the main factor and I dont understand why the one with SAS2 expander and SAS2 hba is performing this bad since we are not even trying to push 1GB/s of throughout total

changing the backplane (expander) is expensive and hard but I can connect the expander to 2x SAS3008 HBA

My question is the limited IOPS is caused by the expander or the newer HBA are much better at handling IOPS ? and based on the answer will just changing the HBA to SAS3008 HBA and keeping the same SAS2 expander will give any benefits ?

Thanks
 

i386

Well-Known Member
Mar 18, 2016
4,221
1,540
113
34
Germany
Hdds are not ideal for small io, the 15k rpm sas disk max out at like 300 iops on the outer tracks and decrease the closer the headers get to the spindle...
 

hatchi

Member
Nov 8, 2014
59
20
8
43
Hdds are not ideal for small io, the 15k rpm sas disk max out at like 300 iops on the outer tracks and decrease the closer the headers get to the spindle...
Correct but this is not the issue
I have exact same setup with same disks , same CPU , motherboard etc
the only diff is the use of expander and different HBAs
and I am getting almost half the IOPS performance without expander than with expander

So my question is if it will help if I upgraded the HBAs to SAS3 HBA keeping the same SAS2 expander ?
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
I guess when you squeeze 24 drives in 4/8 pcie lanes you get a lot of increased latency (6/3 :1 oversubscription basically compared to to the 1:1 map), so I think its possible that the expander is at fault here.

O/c there are also some protocol improvements in SAS3 that might help, which would allow to use the 4/8 lanes from backplane to hba better. You might want to read up on those to see if they seem significant enough.

I guess you can't simply test by swapping the hbas temporarily?
 
  • Like
Reactions: hatchi

hatchi

Member
Nov 8, 2014
59
20
8
43
I guess when you squeeze 24 drives in 4/8 pcie lanes you get a lot of increased latency (6/3 :1 oversubscription basically compared to to the 1:1 map), so I think its possible that the expander is at fault here.

O/c there are also some protocol improvements in SAS3 that might help, which would allow to use the 4/8 lanes from backplane to hba better. You might want to read up on those to see if they seem significant enough.

I guess you can't simply test by swapping the hbas temporarily?
They are both live systems so changing anything need planning
But i think i will build 3rd server just to test but again i will need to run some tests on the server with SAS2 HBAs then repeat same tests with SAS3 HBAs

I know the expander limit throughput but since our application is not usianf that much throuput I thought going with expander and SAS2 HBA will not make any diff
apparently it also limit the IOPS as well which is new to me
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
I think its not IOPS per se but small iops which does not hit the throughput limits.
Similar to networking where many small iops are much harder than few large iops for same throughput since latency is getting in the way.

O/c I am no expert, so take my opinion with a grain of salt (or more)
 
  • Like
Reactions: hatchi