My dual port SAS3 drives also didn't show up as two drives until I added the second SIM cabling. So idk. Wish there was more info on dual port drives and multipath but the info is very sparse.
Dual Port SAS3 SSD's showing in OS as 2 devices
- Thread starter IamSpartacus
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
well that is to be expected - usually you'd have one (or two+ for more bandwith) cable from one SIM to one HBA which is one side of the multipath device, and the same for the second side with the second SIM
Whether it works the way you want should be visible by performance values - my SAS3 drives went up form 1200 to 1600 iirc.
Whether it works the way you want should be visible by performance values - my SAS3 drives went up form 1200 to 1600 iirc.
This is specifically what I am following.
Home · ewwhite/zfs-ha Wiki.
What I summized is that this gives redundancy both on the DAS itself if one SIM was to go bad/offline and the nodes as well
Well that might be true, and apparently it works with the SAS drives ... but it might be the cause of your SATA issues... just saying, not being an expert
But maybe running lsscsi (or the respective vendor tool of your DAS) might provide the discovered layout to see where the various disks are attached to?
But maybe running lsscsi (or the respective vendor tool of your DAS) might provide the discovered layout to see where the various disks are attached to?
This is what I'm getting from my JBOD CLI...
Code:
------------------------------------------------------------------------------
| JB4242 DISKS Modules Status |
------------------------------------------------------------------------------
Disk[00] status: [Present][Enabled] SAS_ADDR : 0x500163600247ea40
Disk[01] status: [Present][Enabled] SAS_ADDR : 0x500163600247ea41
Disk[02] status: [Present][Enabled] SAS_ADDR : 0x500163600247ea42
Disk[03] status: [Present][Enabled] SAS_ADDR : 0x500163600247ea43
Disk[04] status: [Present][Enabled] SAS_ADDR : 0x500163600247ea44
Disk[05] status: [Present][Enabled] SAS_ADDR : 0x500163600247ea45
Disk[06] status: [Present][Enabled] SAS_ADDR : 0x500163600247ea46
Disk[07] status: [Present][Enabled] SAS_ADDR : 0x500163600247ea47
Disk[08] status: [Present][Enabled] SAS_ADDR : 0x500163600247ea48
Disk[09] status: [Present][Enabled] SAS_ADDR : 0x500163600247ea49
Disk[10] status: [Present][Enabled] SAS_ADDR : 0x500163600247ea4a
Disk[11] status: [Present][Enabled] SAS_ADDR : 0x500163600247ea4b
Disk[12] status: [Present][Enabled] SAS_ADDR : 0x500163600247ea4c
Disk[13] status: [Present][Enabled] SAS_ADDR : 0x500163600247ea4d
Disk[14] status: [Present][Enabled] SAS_ADDR : 0x500163600247ea4e
Disk[15] status: [Present][Enabled] SAS_ADDR : 0x500163600247ea4f
Disk[16] status: [Present][Enabled] SAS_ADDR : 0x500163600247ea50
Disk[17] status: [Present][Enabled] SAS_ADDR : 0x500163600247ea51
Disk[18] status: [Present][Enabled] SAS_ADDR : 0x500163600247ea52
Disk[19] status: [Present][Enabled] SAS_ADDR : 0x500163600247ea53
Disk[20] status: [Present][Enabled] SAS_ADDR : 0x500163600247ea54
Disk[21] status: [Present][Enabled] SAS_ADDR : 0x500163600247ea55
Disk[22] status: [Present][Enabled] SAS_ADDR : 0x500163600247ea56
Disk[23] status: [Present][Enabled] SAS_ADDR : 0x500163600247ea57
Disk[24] status: [Present][Enabled] SAS_ADDR : 0x5000cca08c01db21
Disk[25] status: [Present][Enabled] SAS_ADDR : 0x5000cca08c01db4d
Disk[26] status: [Present][Enabled] SAS_ADDR : 0x5000cca08c0027dd
Disk[27] status: [Present][Enabled] SAS_ADDR : 0x5000cca08c01db01
Disk[28] status: [Present][Enabled] SAS_ADDR : 0x5000cca08c01db6d
Disk[29] status: [Present][Enabled] SAS_ADDR : 0x5000cca08c0194ed
Disk[30] status: [Present][Enabled] SAS_ADDR : 0x5000cca08c01d985
Disk[31] status: [Present][Enabled] SAS_ADDR : 0x5000cca08c01d209
------------------------------------------------------------------------------
| JB4242 CPLD Module Status |
------------------------------------------------------------------------------
CPLD Revision : [0x0D]
Self Power Policy: [0x02]
Peer Power Policy: [0x02]
------------------------------------------------------------------------------
| JB4242 State Status |
------------------------------------------------------------------------------
FRU_Error_status : 0x0000000000000008
Self State : [ READY ]
Peer State : [ READY ]
Self Board Revision ID : [0x02]
Peer Board Revision ID : [0x02]
------------------------------------------------------------------------------
Rear Port SAS Address:
SIM0 :
PORT[00] : 500605b00d403658 500605b00d403658 500605b00d403658 500605b00d403658
PORT[01] : 500605b00d403651 500605b00d403651 500605b00d403651 500605b00d403651
PORT[02] : 0000000000000000 0000000000000000 0000000000000000 0000000000000000
PORT[03] : 0000000000000000 0000000000000000 0000000000000000 0000000000000000
SIM1 :
PORT[00] : 500605b00d403659 500605b00d403659 500605b00d403659 500605b00d403659
PORT[01] : 500605b00d403650 500605b00d403650 500605b00d403650 500605b00d403650
PORT[02] : 0000000000000000 0000000000000000 0000000000000000 0000000000000000
PORT[03] : 0000000000000000 0000000000000000 0000000000000000 0000000000000000
Expanders SAS Address:
SIM[00] EXPANDER[00] : 500163600247ea40
SIM[01] EXPANDER[01] : 5001636001bd6300
Code:
JB4242 SIM0 >port
SIM0 :
-------------------------------------------------------
|Port |00|01|02|03|04|05|06|07|08|09|10|11|12|13|14|15|
-------------------------------------------------------
|Type | SAS#0 | SAS#1 | SAS#2 | SAS#3 |
-------------------------------------------------------
|RDY | V| V| V| V| V| V| V| V| | | | | | | | |
-------------------------------------------------------
|SAS | | | | | | | | | | | | | | | | |
-------------------------------------------------------
|SATA | | | | | | | | | | | | | | | | |
-------------------------------------------------------
| 1.5G| | | | | | | | | | | | | | | | |
-------------------------------------------------------
| 3.0G| | | | | | | | | | | | | | | | |
-------------------------------------------------------
| 6.0G| | | | | | | | | | | | | | | | |
-------------------------------------------------------
|12.0G| V| V| V| V| V| V| V| V| | | | | | | | |
-------------------------------------------------------
Port SAS address:
00:500605b00d403658 01:500605b00d403658 02:500605b00d403658 03:500605b00d403658
04:500605b00d403651 05:500605b00d403651 06:500605b00d403651 07:500605b00d403651
08:0000000000000000 09:0000000000000000 10:0000000000000000 11:0000000000000000
12:0000000000000000 13:0000000000000000 14:0000000000000000 15:0000000000000000
SIM1 :
-------------------------------------------------------
|Port |00|01|02|03|04|05|06|07|08|09|10|11|12|13|14|15|
-------------------------------------------------------
|Type | SAS#0 | SAS#1 | SAS#2 | SAS#3 |
-------------------------------------------------------
|RDY | V| V| V| V| V| V| V| V| | | | | | | | |
-------------------------------------------------------
|SAS | | | | | | | | | | | | | | | | |
-------------------------------------------------------
|SATA | | | | | | | | | | | | | | | | |
-------------------------------------------------------
| 1.5G| | | | | | | | | | | | | | | | |
-------------------------------------------------------
| 3.0G| | | | | | | | | | | | | | | | |
-------------------------------------------------------
| 6.0G| | | | | | | | | | | | | | | | |
-------------------------------------------------------
|12.0G| V| V| V| V| V| V| V| V| | | | | | | | |
-------------------------------------------------------
Port SAS address:
00:500605b00d403659 01:500605b00d403659 02:500605b00d403659 03:500605b00d403659
04:500605b00d403650 05:500605b00d403650 06:500605b00d403650 07:500605b00d403650
08:0000000000000000 09:0000000000000000 10:0000000000000000 11:0000000000000000
12:0000000000000000 13:0000000000000000 14:0000000000000000 15:0000000000000000
Code:
JB4242 SIM0 >disks
-------------------------------------------------------------------------------
|Disk |00|01|02|03|04|05|06|07|08|09|10|11|12|13|14|15|16|17|18|19|20|21|22|23|
-------------------------------------------------------------------------------
|PSNT | V| V| V| V| V| V| V| V| V| V| V| V| V| V| V| V| V| V| V| V| V| V| V| V|
-------------------------------------------------------------------------------
|RDY | V| V| V| V| V| V| V| V| V| V| V| V| V| V| V| V| V| V| V| V| V| V| V| V|
-------------------------------------------------------------------------------
|SAS | | | | | | | | | | | | | | | | | | | | | | | | |
-------------------------------------------------------------------------------
|SATA | V| V| V| V| V| V| V| V| V| V| V| V| V| V| V| V| V| V| V| V| V| V| V| V|
-------------------------------------------------------------------------------
| 1.5G| | | | | | | | | | | | | | | | | | | | | | | | |
-------------------------------------------------------------------------------
| 3.0G| | | | | | | | | | | | | | | | | | | | | | | | |
-------------------------------------------------------------------------------
| 6.0G| V| V| V| V| V| V| V| V| V| V| V| V| V| V| V| V| V| V| V| V| V| V| V| V|
-------------------------------------------------------------------------------
|12.0G| | | | | | | | | | | | | | | | | | | | | | | | |
-------------------------------------------------------------------------------
Disk SAS address:
00:500163600247ea40 01:500163600247ea41 02:500163600247ea42 03:500163600247ea43
04:500163600247ea44 05:500163600247ea45 06:500163600247ea46 07:500163600247ea47
08:500163600247ea48 09:500163600247ea49 10:500163600247ea4a 11:500163600247ea4b
12:500163600247ea4c 13:500163600247ea4d 14:500163600247ea4e 15:500163600247ea4f
16:500163600247ea50 17:500163600247ea51 18:500163600247ea52 19:500163600247ea53
20:500163600247ea54 21:500163600247ea55 22:500163600247ea56 23:500163600247ea57
-------------------------------
|SSD |00|01|02|03|04|05|06|07|
-------------------------------
|PSNT | V| V| V| V| V| V| V| V|
-------------------------------
|RDY | V| V| V| V| V| V| V| V|
-------------------------------
|SAS | V| V| V| V| V| V| V| V|
-------------------------------
|SATA | | | | | | | | |
-------------------------------
| 1.5G| | | | | | | | |
-------------------------------
| 3.0G| | | | | | | | |
-------------------------------
| 6.0G| | | | | | | | |
-------------------------------
|12.0G| V| V| V| V| V| V| V| V|
-------------------------------
SSD SAS address:
00:5000cca08c01db21 01:5000cca08c01db4d 02:5000cca08c0027dd 03:5000cca08c01db01
04:5000cca08c01db6d 05:5000cca08c0194ed 06:5000cca08c01d985 07:5000cca08c01d209So it detects them as SATA but provided SAS addresses anyway, and only presents them on one SIM (which is correct).
So I'd assume the OS sees the SAS addresses of the SATA disks twice (due to the dual cableing) and thus maps them as multipath drives... Maybe a lsblk or equivalent OS output might shed some more light on it.
So I'd assume the OS sees the SAS addresses of the SATA disks twice (due to the dual cableing) and thus maps them as multipath drives... Maybe a lsblk or equivalent OS output might shed some more light on it.
Ok so I changed the cabling to remove the second SIM from the equation so that all 4 cables were going from the 9400-16e to the 4 ports in SIM0. No change in behavior what so ever even after clearing all the mpath devices, uninstalling/reinstalling the multipath service, and rebooting multiple times.
When I go down to 2 cables from the controller to the SIM all mpath devices disappear including my dual port SAS3 SSDs.
So it seems the controller is behaving strangely. The problem is I have yet to determine how to get my dual port SAS SSDs to show up properly without all my SATA disks showing up as well.
When I go down to 2 cables from the controller to the SIM all mpath devices disappear including my dual port SAS3 SSDs.
So it seems the controller is behaving strangely. The problem is I have yet to determine how to get my dual port SAS SSDs to show up properly without all my SATA disks showing up as well.
I am going to be adding a second host but as I said, that is just for failover. I did not get the impression that one needed to have two controllers in a single host in order to make use of dual port SAS drives but maybe I just misunderstood what I've been reading.
Found this in the user guide for the controller. Trying to determine exactly what this means but it does look like ports 0 and 1 are considered 1 "SAS Core" and ports 2 and 3 are considered another.
@itronin just gonna tag you here as it'll be easier for context. You are right about the two SAS controllers on the same PCI device as indicated by what I found in the doc above. I have only done the following so far:
Connected all 4 ports of the controller to the same SIM on the JBOD
- With this method I get multipath devices showing up for the SAS SSD's AND all my SATA drives
Connected ports 1&2 to one SIM, ports 3&4 to the second SIM.
- With this method I get multipath devices showing up for the SAS SSD's AND all my SATA drives
Connected ports 1&2 to one SIM
- No multipath devices show up
What I can't figure out is why my SATA drives are showing up as multipath. I guess my JBOD has built in interposers?
Connected all 4 ports of the controller to the same SIM on the JBOD
- With this method I get multipath devices showing up for the SAS SSD's AND all my SATA drives
Connected ports 1&2 to one SIM, ports 3&4 to the second SIM.
- With this method I get multipath devices showing up for the SAS SSD's AND all my SATA drives
Connected ports 1&2 to one SIM
- No multipath devices show up
What I can't figure out is why my SATA drives are showing up as multipath. I guess my JBOD has built in interposers?
Last edited:
The 9300 was only a dual port controller. So obviously it only had one SAS "controller" on it.
just adding this for context. this is not a bad explanation SAS architecture
I'm not sure if "wide ports" are automatically configured or not. You might see if there is a menu item to bundle ports. Since you don't connect with breakouts I'd expect you to bundle each 8644 port to one bundle of 4 or possibly port 1 and 2 in a bundle of 8?
but wide port or not I don't think that would explain what you are seeing...
Is it possible that the expander (each sim) are cross connected to each other? that would explain dual pathing of the sata drives.
would it make sense to take port 1 to one sim and port 2 to another sim? if the sims are cross connected I'd expect your sata drives to show dual path and that might make sense for bundle x4 wide port 1 and x4 wide port 2...
speculating - not really well thought through.
I'm not sure if "wide ports" are automatically configured or not. You might see if there is a menu item to bundle ports. Since you don't connect with breakouts I'd expect you to bundle each 8644 port to one bundle of 4 or possibly port 1 and 2 in a bundle of 8?
but wide port or not I don't think that would explain what you are seeing...
Is it possible that the expander (each sim) are cross connected to each other? that would explain dual pathing of the sata drives.
would it make sense to take port 1 to one sim and port 2 to another sim? if the sims are cross connected I'd expect your sata drives to show dual path and that might make sense for bundle x4 wide port 1 and x4 wide port 2...
speculating - not really well thought through.
I wish I could find detailed info on the QCT JB4242 that shows how it's all wired but I haven't found anything that detailed.
I honestly just care to know if the SATA drives can function as multipath or not so I know whether or not to mount the drives with their multipath device ID or their original single device ID.
What I need to do is test connecting controller port 1 and 3 to the same SIM and see if the SATA drives show up as multipath or not. If they do, that pretty much confirms if the JBOD has interposers.
I honestly just care to know if the SATA drives can function as multipath or not so I know whether or not to mount the drives with their multipath device ID or their original single device ID.
What I need to do is test connecting controller port 1 and 3 to the same SIM and see if the SATA drives show up as multipath or not. If they do, that pretty much confirms if the JBOD has interposers.
Last edited:
another link that might be useful.
I think you may want to test with and possibly run production alternating your -16e connections to your sims.
port 1 -> sim0
port 2 -> sim1
port 3 -> sim0
port 4 -> sim1
It is possible you may also be able to achieve higher throughput if your OS can multipath the SAS IO across multiple ports to each drive. It is also possible though that your OS may elect to use a primary pathway for IO to each device and only use a second pathway if the first is unavailable.
speculating that taking each of your SAS cores and presenting a different pathway may get you that "dual throughput" to your dual ported drive. Assumption being I suppose that the OS driver is preferring a CORE to primary/secondary roles.
regarding interposers (and your pm). I've typically thought of a interposers as a physical card in the drive tray. I suppose that it could be implemented in the backplane and I did just read that the interposers *can* provide multipath support to a SATA drive.
I think you may want to test with and possibly run production alternating your -16e connections to your sims.
port 1 -> sim0
port 2 -> sim1
port 3 -> sim0
port 4 -> sim1
It is possible you may also be able to achieve higher throughput if your OS can multipath the SAS IO across multiple ports to each drive. It is also possible though that your OS may elect to use a primary pathway for IO to each device and only use a second pathway if the first is unavailable.
speculating that taking each of your SAS cores and presenting a different pathway may get you that "dual throughput" to your dual ported drive. Assumption being I suppose that the OS driver is preferring a CORE to primary/secondary roles.
regarding interposers (and your pm). I've typically thought of a interposers as a physical card in the drive tray. I suppose that it could be implemented in the backplane and I did just read that the interposers *can* provide multipath support to a SATA drive.
Yea this is what I'm thinking as well. I'm going to try just port 1 and 3 going to SIM0 first and see how my SATA devices get presented.
The bigger question is how safe do you think it would be to mount the SATA drives using their multipath device ID if the drives continue to be presented to the OS as such?
Indeed I've used that site many times. Did you find a detailed spec doc on the internal components there?