Dual Port SAS3 SSD's showing in OS as 2 devices

Rand__

Well-Known Member
Mar 6, 2014
6,081
1,458
113
Not really, but the manual tells you that the 4 sas ports on each iom are intended for 4 hosts (theoretically one host on 2 ioms each accessing one zone per pair), and that IOM 1 is the secondary (" SIM 1 does not support SATA device status. ")
 
  • Like
Reactions: itronin

IamSpartacus

Well-Known Member
Mar 14, 2016
2,463
615
113
Not really, but the manual tells you that the 4 sas ports on each iom are intended for 4 hosts (theoretically one host on 2 ioms each accessing one zone per pair), and that IOM 1 is the secondary (" SIM 1 does not support SATA device status. ")
Hmmmm, seems odd that they'd want you to be limited to the bandwidth of a single SAS3 connection when the box can hold 32 SAS3 devices and also can cascade. How would they expect to push all that bandwidth over a single connection? I guess the thought is you're only losing at most the 2GB/s (less with the overhead) difference between a single SAS3 connection (6GB/s) and the bandwidth limitation of PCIe 3.0 x8 (8GB/s).
 

Rand__

Well-Known Member
Mar 6, 2014
6,081
1,458
113
Dont forget the old IOPS vs throughput issue... at some point (user density) you want the former more than then latter

But you actually get 2 sas3 connections if you attach IOM0+1
 
  • Like
Reactions: itronin

itronin

Well-Known Member
Nov 24, 2018
891
555
93
Denver, Colorado
The bigger question is how safe do you think it would be to mount the SATA drives using their multipath device ID if the drives continue to be presented to the OS as such?
Generally speaking, I don't "trust it" until I've tested it. I'd use the MP device id's (in linux maybe? IDK for sure, in TNC mounts should use the part UUID right?)

To test it I guess I'd do the following:

Port1 -> sim 0
Port3 -> sim 1

this should be using both pathways and you should see the SATA drives as mp.
Have some "test" traffic running across an array on the sata drives and disconnect Port 1, see what happens. put it back, disconnect port 3 see what happens etc.

reading a bit more I'm not convinced there are integrated interposers in there and am wondering if the sim expanders are cross connected.

bandwidth - old school thinking / design, LFF = spinning rust.
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,463
615
113
Dont forget the old IOPS vs throughput issue... at some point (user density) you want the former more than then latter

But you actually get 2 sas3 connections if you attach IOM0+1
This is true. I just have to do more testing to figure out the way my SATA drives are behaving. I'm a little nervous about mounting the SATA drives using their multipath device ID as I don't want to cause any


Generally speaking, I don't "trust it" until I've tested it. I'd use the MP device id's (in linux maybe? IDK for sure, in TNC mounts should use the part UUID right?)

To test it I guess I'd do the following:

Port1 -> sim 0
Port3 -> sim 1

this should be using both pathways and you should see the SATA drives as mp.
Have some "test" traffic running across an array on the sata drives and disconnect Port 1, see what happens. put it back, disconnect port 3 see what happens etc.

reading a bit more I'm not convinced there are integrated interposers in there and am wondering if the sim expanders are cross connected.

bandwidth - old school thinking / design, LFF = spinning rust.
Yup. Testing is a must. I'll do that this week and report back my findings.
 

itronin

Well-Known Member
Nov 24, 2018
891
555
93
Denver, Colorado
LOL and this gets sooooo confusing trying to think about multipathing and cross connecting expanders. I saw another quanta diagram that showed chaining shelves and the SIMS themselves were cross connected between shelves!
 

itronin

Well-Known Member
Nov 24, 2018
891
555
93
Denver, Colorado
to follow up from the pm thread - LSI 9201-16e/i, LSI 9305-16e/i, LSI 9400-16e/i all appear to share the same physical design pattern which is basically presenting a single pci id device but doubling up the SAS controller cores on the card. In the case of the LSI 9305-24i, I'll guess they tripled up the cores. For reference the 9400-8e/8i refers to the core as a the 3408 and in the 9400-16e/i its referred to as the 3416 however LSI marketing material shows the core for the -16e/i as a 3408 and as the user manuall referred to it then core 0 and core 1 presumably of 3408. Single die per? so two chips or did they cram two 3408's on a single die? dunno - doesn't really matter at that level.
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,463
615
113
to follow up form the pm thread - LSI 9201-16e/i, LSI 9305-16e/i, LSI 9400-16e/i all appear to share the same physical design pattern which is basically presenting a single pci id device but doubling up the SAS controller cores on the card. In the case of the LSI 9305-24i, I'll guess they tripled up the cores. For reference the 9400-8e/8i refers to the core as a the 3408 and in the 9400-16e/i its referred to as the 3416 however LSI marketing material shows the core for the -16e/i as a 3408 and as the user manuall referred to it then core 0 and core 1 presumably of 3408. Single die per? so two chips or did they cram two 3408's on a single die? dunno - doesn't really matter at that level.
Well this just brought another thing to my attention. I hadn't even considered there being two controllers on these boards when I updated the firmware. If there are indeed two different controller cores, then my guess is that storcli would see two controllers and thus I need to update the firmware on the second controller as well to match the first.
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,463
615
113
So I connected 1 cable from controller port 1 and another from controller port 3 to the same SIM and all the drives (including all my SATA drives) are showing up as multipath. So I'll have to do some testing like you said @itronin where I pull one cable at a time while file transfers are in progress to see what kind of behavior I get.

There's definitely a difference between how the SATA disks and SAS disks are being seen by multipath though.

Code:
mpatha (35000cca08c01db20) dm-44 HGST,HBTAC2DH2SUN3.2T
size=2.9T features='0' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 8:0:24:0 sdy     65:128 active ready running
`-+- policy='service-time 0' prio=50 status=enabled
  `- 8:0:57:0 sdbe    67:128 active ready running
 
mpathaa (35000cca266e96620) dm-56 ATA,WDC WD101KRYZ-01
size=9.1T features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 8:0:4:0  sde     8:64   active ready running
`-+- policy='service-time 0' prio=1 status=enabled
  `- 8:0:37:0 sdak    66:64  active ready running
 
Last edited:

Rand__

Well-Known Member
Mar 6, 2014
6,081
1,458
113
Was that difference there before? I mean hwhandler=0 seems pretty indicative...
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,463
615
113
Yea, I just blacklisted all the SATA drives so they no longer show up as multipath.
 

Rand__

Well-Known Member
Mar 6, 2014
6,081
1,458
113
Just remember to do the same on the second box too unless you move the multipath config over as well :)
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,463
615
113
Just remember to do the same on the second box too unless you move the multipath config over as well :)
I intend to move all my disk config files (fstab, crypttab, multipath, etc) over. But thanks for the tip :D.
 

RedX1

Active Member
Aug 11, 2017
111
104
43
Hello



Not necessarily directly related to the thread main topic, but very interesting nevertheless.


While researching some information about the Supermicro BPN-SAS3-216EL1-N4 Backplane for the CSE 216 Chassis, I found this unusually detailed answer on their FAQ’s about maximising performance.


There is some explanation of why sometimes drives are detected in the OS as 2 devices.

1642239848127.png

More detail can be found here.

FAQ Entry | Online Support | Support - Super Micro Computer, Inc.

I hope this is of some help.



Take Care



RedX1