Dual Port SAS3 SSD's showing in OS as 2 devices

IamSpartacus · Dec 29, 2021

This is my first foray into dual port SAS drives with multipath. This may be more suited for the Linux Storage section so a mod can move it if deemed so. I'm currently using a single 9400-16e HBA connected to two SIM's (2 cables to each SIM) on the below DAS unit.

On my Debian 11 server the 8 dual port SAS3 SSD's I'm using are showing up as two different devices as you can see from lsblk. I'm assuming I need to configure something else in the OS to be able to read these drives as single devices to be able to take advantage of the dual port performance they are capable of?

Code:

/dev/sdam      2.9T HGST     HBTAC2DH2SUN3.2T
/dev/sdf       2.9T HGST     HBTAC2DH2SUN3.2T
/dev/sdak      2.9T HGST     HBTAC2DH2SUN3.2T
/dev/sdd       2.9T HGST     HBTAC2DH2SUN3.2T
/dev/sdai      2.9T HGST     HBTAC2DH2SUN3.2T
/dev/sdb       2.9T HGST     HBTAC2DH2SUN3.2T
/dev/sdag      2.9T HGST     HBTAC2DH2SUN3.2T
/dev/sdi       2.9T HGST     HBTAC2DH2SUN3.2T
/dev/sdan      2.9T HGST     HBTAC2DH2SUN3.2T
/dev/sdg       2.9T HGST     HBTAC2DH2SUN3.2T
/dev/sdal      2.9T HGST     HBTAC2DH2SUN3.2T
/dev/sde       2.9T HGST     HBTAC2DH2SUN3.2T
/dev/sdaj      2.9T HGST     HBTAC2DH2SUN3.2T
/dev/sdc       2.9T HGST     HBTAC2DH2SUN3.2T
/dev/sdah      2.9T HGST     HBTAC2DH2SUN3.2T
/dev/sdh       2.9T HGST     HBTAC2DH2SUN3.2T

Rand__ · Dec 29, 2021

I assume something like https://tweenpath.net/install-multipath-debian-server/ ?
To enable multipath for the drives...

IamSpartacus · Dec 29, 2021

Rand__ said:
I assume something like https://tweenpath.net/install-multipath-debian-server/ ?
To enable multipath for the drives...

I was actually JUST about to update my post that I have done this. And it seems to be "working" based on the below output. However I'm mainly concerned with what device ID's to reference when creating a ZFS pool with these drives.

Code:

root@sparta:/dev# multipath -ll
mpatha (35000cca08c01db20) dm-29 HGST,HBTAC2DH2SUN3.2T
size=2.9T features='0' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 8:0:1:0  sdb     8:16   active ready running
`-+- policy='service-time 0' prio=50 status=enabled
  `- 8:0:33:0 sdag    66:0   active ready running
mpathb (35000cca08c01db4c) dm-30 HGST,HBTAC2DH2SUN3.2T
size=2.9T features='0' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 8:0:2:0  sdc     8:32   active ready running
`-+- policy='service-time 0' prio=50 status=enabled
  `- 8:0:34:0 sdah    66:16  active ready running
mpathc (35000cca08c0027dc) dm-31 HGST,HBTAC2DH2SUN3.2T
size=2.9T features='0' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 8:0:3:0  sdd     8:48   active ready running
`-+- policy='service-time 0' prio=50 status=enabled
  `- 8:0:35:0 sdai    66:32  active ready running
mpathd (35000cca08c01db00) dm-32 HGST,HBTAC2DH2SUN3.2T
size=2.9T features='0' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 8:0:4:0  sde     8:64   active ready running
`-+- policy='service-time 0' prio=50 status=enabled
  `- 8:0:36:0 sdaj    66:48  active ready running
mpathe (35000cca08c01db6c) dm-33 HGST,HBTAC2DH2SUN3.2T
size=2.9T features='0' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 8:0:5:0  sdf     8:80   active ready running
`-+- policy='service-time 0' prio=50 status=enabled
  `- 8:0:37:0 sdak    66:64  active ready running
mpathf (35000cca08c0194ec) dm-34 HGST,HBTAC2DH2SUN3.2T
size=2.9T features='0' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 8:0:6:0  sdg     8:96   active ready running
`-+- policy='service-time 0' prio=50 status=enabled
  `- 8:0:38:0 sdal    66:80  active ready running
mpathg (35000cca08c01d984) dm-35 HGST,HBTAC2DH2SUN3.2T
size=2.9T features='0' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 8:0:7:0  sdh     8:112  active ready running
`-+- policy='service-time 0' prio=50 status=enabled
  `- 8:0:39:0 sdam    66:96  active ready running
mpathh (35000cca08c01d208) dm-36 HGST,HBTAC2DH2SUN3.2T
size=2.9T features='0' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 8:0:8:0  sdi     8:128  active ready running
`-+- policy='service-time 0' prio=50 status=enabled
  `- 8:0:40:0 sdan    66:112 active ready running

EDIT: I guess it would be these

Code:

scsi-35000cca08c01db20 -> ../../dm-29
scsi-35000cca08c01db4c -> ../../dm-30
scsi-35000cca08c0027dc -> ../../dm-31
scsi-35000cca08c01db00 -> ../../dm-32
scsi-35000cca08c01db6c -> ../../dm-33
scsi-35000cca08c0194ec -> ../../dm-34
scsi-35000cca08c01d984 -> ../../dm-35
scsi-35000cca08c01d208 -> ../../dm-36

Rand__ · Dec 29, 2021

the normal ones (sdb-i) I'd say

IamSpartacus · Dec 29, 2021

Rand__ said:
the normal ones (sdb-i) I'd say

Well, those ID's change on reboot. Also, the drives show up twice in /dev as you can see in my OP. But I'm thinking the /dev/disk/by-id's I updated my last post with would work.

Rand__ · Dec 29, 2021

Sounds good. Never ran it under debian, only Freebsd

IamSpartacus · Dec 29, 2021

Rand__ said:
Sounds good. Never ran it under debian, only Freebsd

No problem. Referencing the disk's by-id seems to be working. Thanks.

IamSpartacus · Dec 30, 2021

So this is very strange. I just upgraded my server from a 9300-8e to a 9400-16e. Before I had one 8644 connected to each of the 2 SIMs. Now I have two going to each of the SIMs. But what's weird is that now my 14TB SATA3 drives (white label shucked drives) are showing up as multipath? How is that possible?

Code:

mpathp (35000cca259cc0b3c) dm-28 ATA,WDC WD140EDFZ-11
size=13T features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 7:0:24:0 sdx     65:112 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
  `- 7:0:57:0 sdbd    67:112 active ready running
mpathq (35000cca264d555d9) dm-29 ATA,WDC WD140EMFZ-11
size=13T features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 7:0:25:0 sdy     65:128 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
  `- 7:0:58:0 sdbe    67:128 active ready running
mpathr (35000cca264d88ca5) dm-31 ATA,WDC WD140EMFZ-11
size=13T features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 7:0:26:0 sdz     65:144 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
  `- 7:0:59:0 sdbf    67:144 active ready running
mpaths (35000cca264df38bd) dm-33 ATA,WDC WD140EMFZ-11
size=13T features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 7:0:27:0 sdaa    65:160 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
  `- 7:0:60:0 sdbg    67:160 active ready running
mpatht (35000cca264dfbe68) dm-35 ATA,WDC WD140EMFZ-11
size=13T features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 7:0:28:0 sdab    65:176 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
  `- 7:0:61:0 sdbh    67:176 active ready running
mpathv (35000cca264dfe3f3) dm-37 ATA,WDC WD140EMFZ-11
size=13T features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 7:0:29:0 sdac    65:192 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
  `- 7:0:62:0 sdbi    67:192 active ready running
mpathw (35000cca28fc21412) dm-39 ATA,WDC WD140EDFZ-11
size=13T features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 7:0:30:0 sdad    65:208 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
  `- 7:0:63:0 sdbj    67:208 active ready running
mpathx (35000cca28dcf7701) dm-41 ATA,WDC WD140EMFZ-11
size=13T features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 7:0:31:0 sdae    65:224 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
  `- 7:0:64:0 sdbk    67:224 active ready running
mpathy (35000cca28fc0f32f) dm-43 ATA,WDC WD140EDFZ-11
size=13T features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 7:0:32:0 sdaf    65:240 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
  `- 7:0:65:0 sdbl    67:240 active ready running

Rand__ · Dec 31, 2021

It does not seem to properly detect which drives are multipath capable and which not ...

SUSE Documentation

On the SUSE documentation site, find technical documentation, such as quick starts, guides, manuals, and best practices for all SUSE products and solutions.

documentation.suse.com

mentions blacklisting devices, not sure if thats avaialble on your Debian too?

IamSpartacus · Dec 31, 2021

Rand__ said:
It does not seem to properly detect which drives are multipath capable and which not ...

SUSE Documentation

On the SUSE documentation site, find technical documentation, such as quick starts, guides, manuals, and best practices for all SUSE products and solutions.

documentation.suse.com

mentions blacklisting devices, not sure if thats avaialble on your Debian too?

But I guess my question is why is the OS itself detecting the drives as having dual ports because they are detected with two different device IDs. Whether or not they are detected as multi-path is a different issue. Obviously multi-path is detecting them as capable because the OS is reading them as having dual ports.

Rand__ · Dec 31, 2021

The question is what the 9400 is doing differently than the 9300 you used before (if everything else stayed the same)?

You could wap the 9300 back in to see if the multiple devices for Sata remain or not to identify the root change...

IamSpartacus · Dec 31, 2021

Rand__ said:
The question is what the 9400 is doing differently than the 9300 you used before (if everything else stayed the same)?

You could wap the 9300 back in to see if the multiple devices for Sata remain or not to identify the root change...

Well one other variable is the 9300 was already up and booted when I added the second SIM cabling. So it's possible that if I just rebooted the server with the 9300 that the drives would have been detected as multi-path as well. I won't know until I'm able to swap the cards back.

IamSpartacus · Dec 31, 2021

One other thing I thought of was it was possible that this DAS comes with interposers preinstalled but that wouldn't explain why the 10TB drives don't show as dual port and the 14TB drives do.

Rand__ · Dec 31, 2021

maybe it depends on the slot# ? You can try to swap one of the 10 and 14t drives...

What does the DAS's manual say to this (multipathing capabilities)? When i was running a HGST 4U60 it said that Sata drives would only show up on one controller (left iirc). I never tried, but if I attacched 2 hbas on the same (left) controller maybe they would have been seen twice too -> make sure your cableing is correct/to spec of the DAS.

IamSpartacus · Dec 31, 2021

Rand__ said:
maybe it depends on the slot# ? You can try to swap one of the 10 and 14t drives...

What does the DAS's manual say to this (multipathing capabilities)? When i was running a HGST 4U60 it said that Sata drives would only show up on one controller (left iirc). I never tried, but if I attacched 2 hbas on the same (left) controller maybe they would have been seen twice too -> make sure your cableing is correct/to spec of the DAS.

The manual doesn't say much. I have two cables going to one sim and two going to another. All going to the same controller. The controllers in the same slot as the previous controller was.

Rand__ · Dec 31, 2021

Ok, i think we misunderstood each other - what I called controller in my post you called SIM in yours - its in the DAS.

So you have now 4 cables going to the same HBA (the 9400)? That defeats all purposes of multipath (redundancy and or performance), you know?

IamSpartacus · Dec 31, 2021

Rand__ said:
Ok, i think we misunderstood each other - what I called controller in my post you called SIM in yours - its in the DAS.

So you have now 4 cables going to the same HBA (the 9400)? That defeats all purposes of multipath (redundancy and or performance), you know?

I have a second 9400 that's going in a second node. That will be cabled with two more cables going to each SIM.

Rand__ · Dec 31, 2021

so thats for accessing the same data from multiple nodes?

If you only have a single HBA then I don't think you should attach both SIMs to it

IamSpartacus · Dec 31, 2021

Rand__ said:
so thats for accessing the same data from multiple nodes?

If you only have a single HBA then I don't think you should attach both SIMs to it

Its for failover. Only one node will access the data at a time. I was under the impression I needed to connect the same HBA to both SIMs to be able to take advantage of the performance of the dual port drives. I'm essentially trying to mirror this without ZFS.

Home

ZFS High-Availability NAS. Contribute to ewwhite/zfs-ha development by creating an account on GitHub.

github.com

Rand__ · Dec 31, 2021

Yes, i gathered that from the other thread.
I dont think connecting to the same HBA will really work as expected though -
I'd suggest to use two HBAs for that if possible. I know *that* works, not sure if its working you only use one HBA.

But it having both SIMs connected to the same HBA might explain the weird issues you have (albeit not why only half the SATA shows ab in multipath).

Dual Port SAS3 SSD's showing in OS as 2 devices

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member