iSCSI behaving strangely - targets have multiple partitions?

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Perry

Member
Sep 22, 2016
66
11
8
52
[SOLVED]
So the issue seems to be that gparted wasn't showing that old partitions were still there. I was creating the new partitions by using "Unformatted" as the format type, but that seems to leave old cruft behind. I created a new partition that was the entire 54TB volume, using "cleared" as the format type, then deleted that and re-created the partitions using unformatted. lo and behold, the phantom partitions are gone and it seems to work now.



We've been using CentOS 7 for a couple of years now to serve up iSCSI targets to our TigerStore metadata server on our SAN. We've used the same procedure to create the targets, detailed below. I just brought two new arrays online following this procedure and we're seeing something super weird: The first partition of each of the arrays appears in Windows as if it was a single drive with 4 partitions (each array has 4 partitions on the Linux side, but each of those should show up as its own separate iSCSI target). The remaining three partitions are picked up as separate iSCSI targets by the Windows iSCSI initiator, and Windows Disk Management sees them as separate. But the first always shows up as if it was one drive with 4 partitions.

The procedure we use is as follows:
  1. Create RAID 6 in hardware RAID controller
  2. Open each array in gparted and create GPT partition table; create partitions as Primary Partition/Unformatted; Commit changes
  3. reboot
  4. in targetcli, create a block for each partition using /dev/disk/by-partuuid/ for each partition as the unique identifier
  5. in targetcli, create target, pointing to that backstore.
This is a truncated example of what we see in targetcli. (resolve-3 is the first partition on the array, thus the one that's appearing to windows as having 4 partitions. phx-1 is the second partition on that same array, and it appears as its own target, and as a partition inside the target that should be just resolve-3:

Code:
/> ls
o- / ......................................................................................................................... [...]
o- backstores .............................................................................................................. [...]
| o- block ................................................................................................. [Storage Objects: 16]
| | o- phx-1 ......................... [/dev/disk/by-partuuid/9672e724-10bb-4559-91cb-321a3778bb14 (0 bytes) write-thru activated]
| | | o- alua ................................................................................................... [ALUA Groups: 1]
  | | |   o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
| | o- resolve-3 ..................... [/dev/disk/by-partuuid/4be97a07-ad47-4c7a-9ee1-d4630e3c3eb7 (0 bytes) write-thru activated]
| | | o- alua ................................................................................................... [ALUA Groups: 1]
  | | |   o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
| o- fileio ................................................................................................. [Storage Objects: 0]
| o- pscsi .................................................................................................. [Storage Objects: 0]
| o- ramdisk ................................................................................................ [Storage Objects: 0]
o- iscsi ........................................................................................................... [Targets: 16]
| o- iqn.2019-09.com.our-domain-name-here:phx-1 ............................................................................. [TPGs: 1]
| | o- tpg1 .................................................................................................. [gen-acls, no-auth]
| |  o- acls .......................................................................................................... [ACLs: 0]
| |  o- luns .......................................................................................................... [LUNs: 1]
| |  | o- lun0 .................... [block/phx-1 (/dev/disk/by-partuuid/9672e724-10bb-4559-91cb-321a3778bb14) (default_tg_pt_gp)]
| |  o- portals .................................................................................................... [Portals: 1]
| |  o- 10.0.0.1:3260 .................................................................................................... [OK]
| o- iqn.2019-09.com.our-domain-name-here:resolve-3 ......................................................................... [TPGs: 1]
| | o- tpg1 .................................................................................................. [gen-acls, no-auth]
| |  o- acls .......................................................................................................... [ACLs: 0]
| |  o- luns .......................................................................................................... [LUNs: 1]
| |  | o- lun0 ................ [block/resolve-3 (/dev/disk/by-partuuid/4be97a07-ad47-4c7a-9ee1-d4630e3c3eb7) (default_tg_pt_gp)]
| |  o- portals .................................................................................................... [Portals: 1]
| |  o- 10.0.0.1:3260 .................................................................................................... [OK]
 o- loopback ......................................................................................................... [Targets: 0]
o- srpt ............................................................................................................. [Targets: 0]
And this is what Windows sees when the targets are mounted:



Disk 13 is 'phx-1'. Disk 15 should be 'resolve-3' and the size of the first partition it contains is correct for that. The second partition in Disk 15 is 'phx-1'. The two 5.5T partitions also appear as their own drives *and* inside of resolve-3. (the similar looking Disks 14 and 16 are the other array)

I really have no idea what's happening here. The parameters in targetcli are identical to those of the other arrays we're serving up as iSCSI targets, which are working perfectly normally. I've tried wiping out all the partitions, backstores and targets, and rebuilding them from scratch, even with different IQNs in case Windows was caching something.

Any thoughts?
 
Last edited:

Perry

Member
Sep 22, 2016
66
11
8
52
did you configure Windows MPIO?
No. In fact, I believe Tiger wants Multipath turned off.

It can be enabled in the iSCSI initiator when connecting to the target, but the default is off. The thing that's strange is that the only targets exhibiting this behavior are the first ones in the array, on two different arrays, and they're all connected to in the initiator in the same way
 
Last edited:

BoredSysadmin

Not affiliated with Maxell
Mar 2, 2019
1,050
437
83
Well. this behavior seems very much like mpio misconfiguration. The array (if it has dual controllers) presents the same lun twice and if mpio isn't configured - you'd see this.
 

Perry

Member
Sep 22, 2016
66
11
8
52
TigerStore runs on Windows 7 Ultimate. According to them, MPIO isn't even enabled on Win 7 without third party software.

Where would I look to see if the same lun is being presented on the linux side?

The Linux server has two RAID cards, each has two arrays. The arrays where this problem is happening are on the same card. We have had this configuration running for almost a year without issue, but the RAID card died a couple weeks ago, so it has been replaced with an identical card, same firmware, same array size, same configuration on the partitions, though all of it has been rebuilt from scratch (because when the card failed, too much data was corrupted to try to recover it).
 

Perry

Member
Sep 22, 2016
66
11
8
52
So I'm looking in /dev/disk/ to see if there are any conflicts, and I'm not seeing anything.

the devices in question are /dev/sda/* and /dev/sdb/* -- sdc and sdd are the other two arrays on the second controller, and they work perfectly.

/dev/disk/by-id
Code:
[root@localhost by-id]# ls -la
total 0
drwxr-xr-x. 2 root root 720 Sep 24 10:10 .
drwxr-xr-x. 6 root root 120 Sep 24 10:10 ..
lrwxrwxrwx. 1 root root   9 Sep 24 10:10 ata-hp_DVDRAM_GT30L_KZNB4S73118 -> ../../sr0
lrwxrwxrwx. 1 root root   9 Sep 24 13:40 ata-SSD2SC240G1CS11754D117-514_PNY52150000099990174 -> ../../sde
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 ata-SSD2SC240G1CS11754D117-514_PNY52150000099990174-part1 -> ../../sde1
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 ata-SSD2SC240G1CS11754D117-514_PNY52150000099990174-part2 -> ../../sde2
lrwxrwxrwx. 1 root root  10 Sep 24 10:10 dm-name-centos-home -> ../../dm-2
lrwxrwxrwx. 1 root root  10 Sep 24 10:10 dm-name-centos-root -> ../../dm-0
lrwxrwxrwx. 1 root root  10 Sep 24 10:10 dm-name-centos-swap -> ../../dm-1
lrwxrwxrwx. 1 root root  10 Sep 24 10:10 dm-uuid-LVM-kqUDMYvsD3PJBnY03EWtQCa0POEbrrx7d4IqBZFLxNWc7OFHZkH4w6sFkZcAMpr7 -> ../../dm-0
lrwxrwxrwx. 1 root root  10 Sep 24 10:10 dm-uuid-LVM-kqUDMYvsD3PJBnY03EWtQCa0POEbrrx7jggyOBttR4BjNMD0BPCbNNYuQd4LZfne -> ../../dm-2
lrwxrwxrwx. 1 root root  10 Sep 24 10:10 dm-uuid-LVM-kqUDMYvsD3PJBnY03EWtQCa0POEbrrx7v3mZpDKah4H9CApb9ivdcmSKnngPLF00 -> ../../dm-1
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 lvm-pv-uuid-NdiWd8-Zx3f-rLll-NFhF-qjAF-e5wp-y9c4w4 -> ../../sde2
lrwxrwxrwx. 1 root root   9 Sep 24 13:40 scsi-232180f2600d00000 -> ../../sda
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 scsi-232180f2600d00000-part1 -> ../../sda1
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 scsi-232180f2600d00000-part2 -> ../../sda2
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 scsi-232180f2600d00000-part3 -> ../../sda3
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 scsi-232180f2600d00000-part4 -> ../../sda4
lrwxrwxrwx. 1 root root   9 Sep 24 13:40 scsi-2aa984cb500d00000 -> ../../sdc
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 scsi-2aa984cb500d00000-part1 -> ../../sdc1
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 scsi-2aa984cb500d00000-part2 -> ../../sdc2
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 scsi-2aa984cb500d00000-part3 -> ../../sdc3
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 scsi-2aa984cb500d00000-part4 -> ../../sdc4
lrwxrwxrwx. 1 root root   9 Sep 24 13:40 scsi-2aaa8101b00d00000 -> ../../sdd
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 scsi-2aaa8101b00d00000-part1 -> ../../sdd1
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 scsi-2aaa8101b00d00000-part2 -> ../../sdd2
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 scsi-2aaa8101b00d00000-part3 -> ../../sdd3
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 scsi-2aaa8101b00d00000-part4 -> ../../sdd4
lrwxrwxrwx. 1 root root   9 Sep 24 13:40 scsi-2b248420a00d00000 -> ../../sdb
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 scsi-2b248420a00d00000-part1 -> ../../sdb1
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 scsi-2b248420a00d00000-part2 -> ../../sdb2
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 scsi-2b248420a00d00000-part3 -> ../../sdb3
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 scsi-2b248420a00d00000-part4 -> ../../sdb4
lrwxrwxrwx. 1 root root   9 Sep 24 13:40 wwn-0x5f8db4c525990174 -> ../../sde
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 wwn-0x5f8db4c525990174-part1 -> ../../sde1
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 wwn-0x5f8db4c525990174-part2 -> ../../sde2

/dev/disk/by-partuuid (this is how we're referencing the partitions in targetcli)
Code:
[root@localhost by-partuuid]# ls -la
total 0
drwxr-xr-x. 2 root root 360 Sep 24 10:10 .
drwxr-xr-x. 6 root root 120 Sep 24 10:10 ..
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 0c04f368-b698-461c-8bfc-b416479e2634 -> ../../sda3
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 2693949f-15c5-4c39-b7e5-578658dd05d8 -> ../../sdd1
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 29cea866-0aff-45ab-9577-7d8da19b346e -> ../../sdb3
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 2b997020-88c9-4c1e-bf77-6e06132b1506 -> ../../sdd2
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 3856dff1-75af-4462-bf58-4412e0dd61fe -> ../../sdd3
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 4672c917-01ba-4f02-85dd-e53c9d41f7fa -> ../../sdc1
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 4be97a07-ad47-4c7a-9ee1-d4630e3c3eb7 -> ../../sda1
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 7aedcbfc-3bb4-4688-90ed-6b9ddfa0d58d -> ../../sdb2
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 9672e724-10bb-4559-91cb-321a3778bb14 -> ../../sda2
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 a954e2e5-6708-4d70-bb05-a7539cb280f1 -> ../../sdb1
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 c63f539d-d230-4744-901b-1296f3a26727 -> ../../sda4
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 c8b6e3b9-7ee0-4553-9ec3-78e3d82137a4 -> ../../sdc2
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 cb31ea47-3263-4333-ac87-7c7f7eb358df -> ../../sdc4
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 dc7ae8be-3726-454b-8c87-2dd7ee08a924 -> ../../sdc3
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 ecaff211-b357-4c0f-9c2c-ae66f56be457 -> ../../sdb4
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 f201e565-99c9-4bf4-871c-67744d283730 -> ../../sdd4


/dev/disk/by-path
Code:
[root@localhost by-path]# ls -la
total 0
drwxr-xr-x. 2 root root 520 Sep 24 10:10 .
drwxr-xr-x. 6 root root 120 Sep 24 10:10 ..
lrwxrwxrwx. 1 root root   9 Sep 24 13:40 pci-0000:00:1f.2-ata-5.0 -> ../../sde
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 pci-0000:00:1f.2-ata-5.0-part1 -> ../../sde1
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 pci-0000:00:1f.2-ata-5.0-part2 -> ../../sde2
lrwxrwxrwx. 1 root root   9 Sep 24 10:10 pci-0000:00:1f.2-ata-6.0 -> ../../sr0
lrwxrwxrwx. 1 root root   9 Sep 24 13:40 pci-0000:02:00.0-scsi-0:0:0:0 -> ../../sda
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 pci-0000:02:00.0-scsi-0:0:0:0-part1 -> ../../sda1
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 pci-0000:02:00.0-scsi-0:0:0:0-part2 -> ../../sda2
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 pci-0000:02:00.0-scsi-0:0:0:0-part3 -> ../../sda3
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 pci-0000:02:00.0-scsi-0:0:0:0-part4 -> ../../sda4
lrwxrwxrwx. 1 root root   9 Sep 24 13:40 pci-0000:02:00.0-scsi-0:0:1:0 -> ../../sdb
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 pci-0000:02:00.0-scsi-0:0:1:0-part1 -> ../../sdb1
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 pci-0000:02:00.0-scsi-0:0:1:0-part2 -> ../../sdb2
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 pci-0000:02:00.0-scsi-0:0:1:0-part3 -> ../../sdb3
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 pci-0000:02:00.0-scsi-0:0:1:0-part4 -> ../../sdb4
lrwxrwxrwx. 1 root root   9 Sep 24 13:40 pci-0000:03:00.0-scsi-0:0:0:0 -> ../../sdc
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 pci-0000:03:00.0-scsi-0:0:0:0-part1 -> ../../sdc1
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 pci-0000:03:00.0-scsi-0:0:0:0-part2 -> ../../sdc2
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 pci-0000:03:00.0-scsi-0:0:0:0-part3 -> ../../sdc3
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 pci-0000:03:00.0-scsi-0:0:0:0-part4 -> ../../sdc4
lrwxrwxrwx. 1 root root   9 Sep 24 13:40 pci-0000:03:00.0-scsi-0:0:1:0 -> ../../sdd
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 pci-0000:03:00.0-scsi-0:0:1:0-part1 -> ../../sdd1
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 pci-0000:03:00.0-scsi-0:0:1:0-part2 -> ../../sdd2
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 pci-0000:03:00.0-scsi-0:0:1:0-part3 -> ../../sdd3
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 pci-0000:03:00.0-scsi-0:0:1:0-part4 -> ../../sdd4



/dev/disk/by-uuid
Code:
[root@localhost by-uuid]# ls -la
total 0
drwxr-xr-x. 2 root root 120 Sep 24 10:10 .
drwxr-xr-x. 6 root root 120 Sep 24 10:10 ..
lrwxrwxrwx. 1 root root  10 Sep 24 10:10 15502447-19d0-4761-9e88-535d074f6f5b -> ../../dm-1
lrwxrwxrwx. 1 root root  10 Sep 24 10:10 7d2acd30-4ce7-400e-b5fb-35d95e20d193 -> ../../dm-0
lrwxrwxrwx. 1 root root  10 Sep 24 13:40 899c1871-d6c6-4ffe-b045-6f9146369494 -> ../../sde1
lrwxrwxrwx. 1 root root  10 Sep 24 10:10 d3b1fb67-fd64-43e8-a470-49db464f53ce -> ../../dm-2
 
Last edited:

Perry

Member
Sep 22, 2016
66
11
8
52
Fixed the problem - it was a partition issue on the linux side. See note at the top of the original post.