iSCSI behaving strangely - targets have multiple partitions?

Discussion in 'Linux Admins, Storage and Virtualization' started by Perry, Sep 24, 2019.

  1. Perry

    Perry Member

    Joined:
    Sep 22, 2016
    Messages:
    45
    Likes Received:
    3
    [SOLVED]
    So the issue seems to be that gparted wasn't showing that old partitions were still there. I was creating the new partitions by using "Unformatted" as the format type, but that seems to leave old cruft behind. I created a new partition that was the entire 54TB volume, using "cleared" as the format type, then deleted that and re-created the partitions using unformatted. lo and behold, the phantom partitions are gone and it seems to work now.



    We've been using CentOS 7 for a couple of years now to serve up iSCSI targets to our TigerStore metadata server on our SAN. We've used the same procedure to create the targets, detailed below. I just brought two new arrays online following this procedure and we're seeing something super weird: The first partition of each of the arrays appears in Windows as if it was a single drive with 4 partitions (each array has 4 partitions on the Linux side, but each of those should show up as its own separate iSCSI target). The remaining three partitions are picked up as separate iSCSI targets by the Windows iSCSI initiator, and Windows Disk Management sees them as separate. But the first always shows up as if it was one drive with 4 partitions.

    The procedure we use is as follows:
    1. Create RAID 6 in hardware RAID controller
    2. Open each array in gparted and create GPT partition table; create partitions as Primary Partition/Unformatted; Commit changes
    3. reboot
    4. in targetcli, create a block for each partition using /dev/disk/by-partuuid/ for each partition as the unique identifier
    5. in targetcli, create target, pointing to that backstore.
    This is a truncated example of what we see in targetcli. (resolve-3 is the first partition on the array, thus the one that's appearing to windows as having 4 partitions. phx-1 is the second partition on that same array, and it appears as its own target, and as a partition inside the target that should be just resolve-3:

    Code:
    /> ls
    o- / ......................................................................................................................... [...]
    o- backstores .............................................................................................................. [...]
    | o- block ................................................................................................. [Storage Objects: 16]
    | | o- phx-1 ......................... [/dev/disk/by-partuuid/9672e724-10bb-4559-91cb-321a3778bb14 (0 bytes) write-thru activated]
    | | | o- alua ................................................................................................... [ALUA Groups: 1]
      | | |   o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
    | | o- resolve-3 ..................... [/dev/disk/by-partuuid/4be97a07-ad47-4c7a-9ee1-d4630e3c3eb7 (0 bytes) write-thru activated]
    | | | o- alua ................................................................................................... [ALUA Groups: 1]
      | | |   o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
    | o- fileio ................................................................................................. [Storage Objects: 0]
    | o- pscsi .................................................................................................. [Storage Objects: 0]
    | o- ramdisk ................................................................................................ [Storage Objects: 0]
    o- iscsi ........................................................................................................... [Targets: 16]
    | o- iqn.2019-09.com.our-domain-name-here:phx-1 ............................................................................. [TPGs: 1]
    | | o- tpg1 .................................................................................................. [gen-acls, no-auth]
    | |  o- acls .......................................................................................................... [ACLs: 0]
    | |  o- luns .......................................................................................................... [LUNs: 1]
    | |  | o- lun0 .................... [block/phx-1 (/dev/disk/by-partuuid/9672e724-10bb-4559-91cb-321a3778bb14) (default_tg_pt_gp)]
    | |  o- portals .................................................................................................... [Portals: 1]
    | |  o- 10.0.0.1:3260 .................................................................................................... [OK]
    | o- iqn.2019-09.com.our-domain-name-here:resolve-3 ......................................................................... [TPGs: 1]
    | | o- tpg1 .................................................................................................. [gen-acls, no-auth]
    | |  o- acls .......................................................................................................... [ACLs: 0]
    | |  o- luns .......................................................................................................... [LUNs: 1]
    | |  | o- lun0 ................ [block/resolve-3 (/dev/disk/by-partuuid/4be97a07-ad47-4c7a-9ee1-d4630e3c3eb7) (default_tg_pt_gp)]
    | |  o- portals .................................................................................................... [Portals: 1]
    | |  o- 10.0.0.1:3260 .................................................................................................... [OK]
     o- loopback ......................................................................................................... [Targets: 0]
    o- srpt ............................................................................................................. [Targets: 0]
    And this is what Windows sees when the targets are mounted:

    [​IMG]

    Disk 13 is 'phx-1'. Disk 15 should be 'resolve-3' and the size of the first partition it contains is correct for that. The second partition in Disk 15 is 'phx-1'. The two 5.5T partitions also appear as their own drives *and* inside of resolve-3. (the similar looking Disks 14 and 16 are the other array)

    I really have no idea what's happening here. The parameters in targetcli are identical to those of the other arrays we're serving up as iSCSI targets, which are working perfectly normally. I've tried wiping out all the partitions, backstores and targets, and rebuilding them from scratch, even with different IQNs in case Windows was caching something.

    Any thoughts?
     
    #1
    Last edited: Sep 24, 2019
  2. BoredSysadmin

    BoredSysadmin Active Member

    Joined:
    Mar 2, 2019
    Messages:
    281
    Likes Received:
    62
    did you configure Windows MPIO?
     
    #2
  3. Perry

    Perry Member

    Joined:
    Sep 22, 2016
    Messages:
    45
    Likes Received:
    3
    No. In fact, I believe Tiger wants Multipath turned off.

    It can be enabled in the iSCSI initiator when connecting to the target, but the default is off. The thing that's strange is that the only targets exhibiting this behavior are the first ones in the array, on two different arrays, and they're all connected to in the initiator in the same way
     
    #3
    Last edited: Sep 24, 2019
  4. BoredSysadmin

    BoredSysadmin Active Member

    Joined:
    Mar 2, 2019
    Messages:
    281
    Likes Received:
    62
    Well. this behavior seems very much like mpio misconfiguration. The array (if it has dual controllers) presents the same lun twice and if mpio isn't configured - you'd see this.
     
    #4
  5. Perry

    Perry Member

    Joined:
    Sep 22, 2016
    Messages:
    45
    Likes Received:
    3
    TigerStore runs on Windows 7 Ultimate. According to them, MPIO isn't even enabled on Win 7 without third party software.

    Where would I look to see if the same lun is being presented on the linux side?

    The Linux server has two RAID cards, each has two arrays. The arrays where this problem is happening are on the same card. We have had this configuration running for almost a year without issue, but the RAID card died a couple weeks ago, so it has been replaced with an identical card, same firmware, same array size, same configuration on the partitions, though all of it has been rebuilt from scratch (because when the card failed, too much data was corrupted to try to recover it).
     
    #5
  6. Perry

    Perry Member

    Joined:
    Sep 22, 2016
    Messages:
    45
    Likes Received:
    3
    So I'm looking in /dev/disk/ to see if there are any conflicts, and I'm not seeing anything.

    the devices in question are /dev/sda/* and /dev/sdb/* -- sdc and sdd are the other two arrays on the second controller, and they work perfectly.

    /dev/disk/by-id
    Code:
    [root@localhost by-id]# ls -la
    total 0
    drwxr-xr-x. 2 root root 720 Sep 24 10:10 .
    drwxr-xr-x. 6 root root 120 Sep 24 10:10 ..
    lrwxrwxrwx. 1 root root   9 Sep 24 10:10 ata-hp_DVDRAM_GT30L_KZNB4S73118 -> ../../sr0
    lrwxrwxrwx. 1 root root   9 Sep 24 13:40 ata-SSD2SC240G1CS11754D117-514_PNY52150000099990174 -> ../../sde
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 ata-SSD2SC240G1CS11754D117-514_PNY52150000099990174-part1 -> ../../sde1
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 ata-SSD2SC240G1CS11754D117-514_PNY52150000099990174-part2 -> ../../sde2
    lrwxrwxrwx. 1 root root  10 Sep 24 10:10 dm-name-centos-home -> ../../dm-2
    lrwxrwxrwx. 1 root root  10 Sep 24 10:10 dm-name-centos-root -> ../../dm-0
    lrwxrwxrwx. 1 root root  10 Sep 24 10:10 dm-name-centos-swap -> ../../dm-1
    lrwxrwxrwx. 1 root root  10 Sep 24 10:10 dm-uuid-LVM-kqUDMYvsD3PJBnY03EWtQCa0POEbrrx7d4IqBZFLxNWc7OFHZkH4w6sFkZcAMpr7 -> ../../dm-0
    lrwxrwxrwx. 1 root root  10 Sep 24 10:10 dm-uuid-LVM-kqUDMYvsD3PJBnY03EWtQCa0POEbrrx7jggyOBttR4BjNMD0BPCbNNYuQd4LZfne -> ../../dm-2
    lrwxrwxrwx. 1 root root  10 Sep 24 10:10 dm-uuid-LVM-kqUDMYvsD3PJBnY03EWtQCa0POEbrrx7v3mZpDKah4H9CApb9ivdcmSKnngPLF00 -> ../../dm-1
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 lvm-pv-uuid-NdiWd8-Zx3f-rLll-NFhF-qjAF-e5wp-y9c4w4 -> ../../sde2
    lrwxrwxrwx. 1 root root   9 Sep 24 13:40 scsi-232180f2600d00000 -> ../../sda
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 scsi-232180f2600d00000-part1 -> ../../sda1
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 scsi-232180f2600d00000-part2 -> ../../sda2
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 scsi-232180f2600d00000-part3 -> ../../sda3
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 scsi-232180f2600d00000-part4 -> ../../sda4
    lrwxrwxrwx. 1 root root   9 Sep 24 13:40 scsi-2aa984cb500d00000 -> ../../sdc
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 scsi-2aa984cb500d00000-part1 -> ../../sdc1
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 scsi-2aa984cb500d00000-part2 -> ../../sdc2
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 scsi-2aa984cb500d00000-part3 -> ../../sdc3
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 scsi-2aa984cb500d00000-part4 -> ../../sdc4
    lrwxrwxrwx. 1 root root   9 Sep 24 13:40 scsi-2aaa8101b00d00000 -> ../../sdd
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 scsi-2aaa8101b00d00000-part1 -> ../../sdd1
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 scsi-2aaa8101b00d00000-part2 -> ../../sdd2
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 scsi-2aaa8101b00d00000-part3 -> ../../sdd3
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 scsi-2aaa8101b00d00000-part4 -> ../../sdd4
    lrwxrwxrwx. 1 root root   9 Sep 24 13:40 scsi-2b248420a00d00000 -> ../../sdb
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 scsi-2b248420a00d00000-part1 -> ../../sdb1
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 scsi-2b248420a00d00000-part2 -> ../../sdb2
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 scsi-2b248420a00d00000-part3 -> ../../sdb3
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 scsi-2b248420a00d00000-part4 -> ../../sdb4
    lrwxrwxrwx. 1 root root   9 Sep 24 13:40 wwn-0x5f8db4c525990174 -> ../../sde
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 wwn-0x5f8db4c525990174-part1 -> ../../sde1
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 wwn-0x5f8db4c525990174-part2 -> ../../sde2
    

    /dev/disk/by-partuuid (this is how we're referencing the partitions in targetcli)
    Code:
    [root@localhost by-partuuid]# ls -la
    total 0
    drwxr-xr-x. 2 root root 360 Sep 24 10:10 .
    drwxr-xr-x. 6 root root 120 Sep 24 10:10 ..
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 0c04f368-b698-461c-8bfc-b416479e2634 -> ../../sda3
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 2693949f-15c5-4c39-b7e5-578658dd05d8 -> ../../sdd1
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 29cea866-0aff-45ab-9577-7d8da19b346e -> ../../sdb3
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 2b997020-88c9-4c1e-bf77-6e06132b1506 -> ../../sdd2
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 3856dff1-75af-4462-bf58-4412e0dd61fe -> ../../sdd3
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 4672c917-01ba-4f02-85dd-e53c9d41f7fa -> ../../sdc1
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 4be97a07-ad47-4c7a-9ee1-d4630e3c3eb7 -> ../../sda1
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 7aedcbfc-3bb4-4688-90ed-6b9ddfa0d58d -> ../../sdb2
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 9672e724-10bb-4559-91cb-321a3778bb14 -> ../../sda2
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 a954e2e5-6708-4d70-bb05-a7539cb280f1 -> ../../sdb1
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 c63f539d-d230-4744-901b-1296f3a26727 -> ../../sda4
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 c8b6e3b9-7ee0-4553-9ec3-78e3d82137a4 -> ../../sdc2
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 cb31ea47-3263-4333-ac87-7c7f7eb358df -> ../../sdc4
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 dc7ae8be-3726-454b-8c87-2dd7ee08a924 -> ../../sdc3
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 ecaff211-b357-4c0f-9c2c-ae66f56be457 -> ../../sdb4
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 f201e565-99c9-4bf4-871c-67744d283730 -> ../../sdd4


    /dev/disk/by-path
    Code:
    [root@localhost by-path]# ls -la
    total 0
    drwxr-xr-x. 2 root root 520 Sep 24 10:10 .
    drwxr-xr-x. 6 root root 120 Sep 24 10:10 ..
    lrwxrwxrwx. 1 root root   9 Sep 24 13:40 pci-0000:00:1f.2-ata-5.0 -> ../../sde
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 pci-0000:00:1f.2-ata-5.0-part1 -> ../../sde1
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 pci-0000:00:1f.2-ata-5.0-part2 -> ../../sde2
    lrwxrwxrwx. 1 root root   9 Sep 24 10:10 pci-0000:00:1f.2-ata-6.0 -> ../../sr0
    lrwxrwxrwx. 1 root root   9 Sep 24 13:40 pci-0000:02:00.0-scsi-0:0:0:0 -> ../../sda
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 pci-0000:02:00.0-scsi-0:0:0:0-part1 -> ../../sda1
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 pci-0000:02:00.0-scsi-0:0:0:0-part2 -> ../../sda2
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 pci-0000:02:00.0-scsi-0:0:0:0-part3 -> ../../sda3
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 pci-0000:02:00.0-scsi-0:0:0:0-part4 -> ../../sda4
    lrwxrwxrwx. 1 root root   9 Sep 24 13:40 pci-0000:02:00.0-scsi-0:0:1:0 -> ../../sdb
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 pci-0000:02:00.0-scsi-0:0:1:0-part1 -> ../../sdb1
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 pci-0000:02:00.0-scsi-0:0:1:0-part2 -> ../../sdb2
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 pci-0000:02:00.0-scsi-0:0:1:0-part3 -> ../../sdb3
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 pci-0000:02:00.0-scsi-0:0:1:0-part4 -> ../../sdb4
    lrwxrwxrwx. 1 root root   9 Sep 24 13:40 pci-0000:03:00.0-scsi-0:0:0:0 -> ../../sdc
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 pci-0000:03:00.0-scsi-0:0:0:0-part1 -> ../../sdc1
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 pci-0000:03:00.0-scsi-0:0:0:0-part2 -> ../../sdc2
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 pci-0000:03:00.0-scsi-0:0:0:0-part3 -> ../../sdc3
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 pci-0000:03:00.0-scsi-0:0:0:0-part4 -> ../../sdc4
    lrwxrwxrwx. 1 root root   9 Sep 24 13:40 pci-0000:03:00.0-scsi-0:0:1:0 -> ../../sdd
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 pci-0000:03:00.0-scsi-0:0:1:0-part1 -> ../../sdd1
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 pci-0000:03:00.0-scsi-0:0:1:0-part2 -> ../../sdd2
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 pci-0000:03:00.0-scsi-0:0:1:0-part3 -> ../../sdd3
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 pci-0000:03:00.0-scsi-0:0:1:0-part4 -> ../../sdd4



    /dev/disk/by-uuid
    Code:
    [root@localhost by-uuid]# ls -la
    total 0
    drwxr-xr-x. 2 root root 120 Sep 24 10:10 .
    drwxr-xr-x. 6 root root 120 Sep 24 10:10 ..
    lrwxrwxrwx. 1 root root  10 Sep 24 10:10 15502447-19d0-4761-9e88-535d074f6f5b -> ../../dm-1
    lrwxrwxrwx. 1 root root  10 Sep 24 10:10 7d2acd30-4ce7-400e-b5fb-35d95e20d193 -> ../../dm-0
    lrwxrwxrwx. 1 root root  10 Sep 24 13:40 899c1871-d6c6-4ffe-b045-6f9146369494 -> ../../sde1
    lrwxrwxrwx. 1 root root  10 Sep 24 10:10 d3b1fb67-fd64-43e8-a470-49db464f53ce -> ../../dm-2
    
    
     
    #6
    Last edited: Sep 24, 2019
  7. Perry

    Perry Member

    Joined:
    Sep 22, 2016
    Messages:
    45
    Likes Received:
    3
    Fixed the problem - it was a partition issue on the linux side. See note at the top of the original post.
     
    #7
Similar Threads: iSCSI behaving
Forum Title Date
Linux Admins, Storage and Virtualization NFS / iSCSI security on a compromised Initiator (client) Feb 5, 2019
Linux Admins, Storage and Virtualization Ceph iSCSI Target with Proxmox VE implementation ? Jul 18, 2018
Linux Admins, Storage and Virtualization PETASAN HA iSCSI Opensource free solution with CEPH Mar 4, 2018
Linux Admins, Storage and Virtualization Only seeing some of iSCSI target from Windows 7 Jan 4, 2018
Linux Admins, Storage and Virtualization A decent iSCSI-target VM with UNMAP/DISCARD support Apr 10, 2017

Share This Page