NVMe devices and CentOS Persistent Naming

[handruin]

Hopefully I'm in the right area to be asking this question if not please feel free to move this post if appropriate.

I'm researching a bit to understand Persistent Naming under CentOS 6.7 with NVMe storage devices. As of right now I have two Intel P3700 devices in a system and only one of them shows as having a persistent name. I'd like to make sure the device path doesn't change when I use one (or both) of these in a zpool. Is there something specific I should be configuring to make this consistent such as creating my own udev rule? As of right now my base install has no storage rules defined under the udev rules.d so I don't know if this is part of my problem? I am unclear how I'd have even one NVMe devices mapped with a persistent name and not the other.

Any thoughts on where I can look into this further or do I even need to? I'm going to try and increase the debug logging and trigger udev to see if there are any issues there.

[root@lenovo3650M5-DRBD0 ~]# ls -1 /etc/udev/rules.d/
51-dlm.rules
60-fprint-autosuspend.rules
60-pcmcia.rules
60-raw.rules
70-persistent-cd.rules
70-persistent-net.rules
90-alsa.rules
90-hal.rules
98-kexec.rules

[root@lenovo3650M5-DRBD0 ~]# uname -a
Linux lenovo3650M5-DRBD0 2.6.32-573.3.1.el6.x86_64 #1 SMP Thu Aug 13 22:55:16 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

[root@lenovo3650M5-DRBD0 ~]# ls -al /dev/nvme*n*
brw-rw----. 1 root disk 259, 0 Mar 28 18:08 /dev/nvme0n1
brw-rw----. 1 root disk 259, 1 Mar 28 18:08 /dev/nvme1n1

[root@lenovo3650M5-DRBD0 ~]# ls -al /dev/disk/by-id/
total 0
drwxr-xr-x. 2 root root 540 Mar 28 18:08 .
drwxr-xr-x. 5 root root 100 Mar 28 15:52 ..
lrwxrwxrwx. 1 root root  13 Mar 28 18:08 wwn-0x65cd2e4080864356494e000000010000 -> ../../nvme1n1

(note: I removed other non NVMe devices for ease of readability)

 

[cperalt1]

wwn-0x65cd2e4080864356494e000000010000

You want to create your pool using the UUID as the other name can change at anytime. Good practice is to create zpool with UUID and the zpool export and import to ensure that it will mount correctly. The UUID is consistent versus name by path is not.

 

[handruin]

wwn-0x65cd2e4080864356494e000000010000

You want to create your pool using the UUID as the other name can change at anytime. Good practice is to create zpool with UUID and the zpool export and import to ensure that it will mount correctly. The UUID is consistent versus name by path is not.

There is no "/dev/disk/by-uuid/" generated under CentOS 6.7 for nvme devices. Even if I run "blkid /dev/nvme0n1" nothing is returned by the drive. After digging into this more with my coworker, he discovered that for some reason all nvme devices inside the system return the same WWN value when querying page 83 and this is the reason why we're seeing only one device linked with a wwn. The udev rule must be overwriting each device because the wwn is the same.

scsi_id -p 0x83 --whitelisted -d /dev/nvme0n1

As a temp workaround, the proposal is to use our own udev rule for persistent storage of the following under "/etc/udev/rules.d/61-persistent-storage.rules":

KERNEL=="nvme*", ENV{ID_SERIAL}!="?*", IMPORT{program}="scsi_id --export --whitelisted -d $tempnode"

KERNEL=="nvme*", ENV{ID_SCSI_SERIAL}=="?*", SYMLINK+="disk/by-id/nvme-$env{ID_SCSI_SERIAL}"

KERNEL=="nvme*", ENV{DEVTYPE}=="partition", SYMLINK+="disk/by-id/nvme-$env{ID_SCSI_SERIAL}-part%n"

This maps each nvme device using the drive's serial number since we are not able to get a unique WWN from them.

[root@lenovo3650M5-DRBD0 nvme-cli]# ls -al /dev/disk/by-id/
total 0
drwxr-xr-x. 2 root root 580 Mar 29 14:51 .
drwxr-xr-x. 5 root root 100 Mar 28 15:52 ..
lrwxrwxrwx. 1 root root  13 Mar 29 14:51 nvme-CVFT54400046800HGN -> ../../nvme0n1
lrwxrwxrwx. 1 root root  13 Mar 29 14:50 nvme-CVFT544000B1800HGN -> ../../nvme1n1