All my data gone on OMV ZFS NAS///HELP!

[scrummie02]

My ZFS filesystems are there and everything but all my data is just gone. I see my exports and configuration but everything in my ZFS datasets have nothing in them at all, nothing.....



How could all of my data just disappear from my ZFS filesystem with nothing in it?

I see the following:

NAME USED AVAIL REFER MOUNTPOINT

data 6.17T 4.07T 6.17T /data

But when moving to the directory and doing LS nothing shows up! Where is my data?!?!?!

 

[scrummie02]

Here is the output of my zpool

ool status (zpool status):

  pool: data
 state: ONLINE
  scan: scrub repaired 0B in 0 days 09:59:26 with 0 errors on Sun Jun 14 10:23:32 2020
config:

    NAME                                          STATE     READ WRITE CKSUM
    data                                          ONLINE       0     0     0
      raidz1-0                                    ONLINE       0     0     0
        ata-WDC_WD40EFRX-68N32N0_WD-WCC7K4HFZS6D  ONLINE       0     0     0
        ata-WDC_WD40EFRX-68N32N0_WD-WCC7K5KFP3ST  ONLINE       0     0     0
        ata-WDC_WD40EFRX-68N32N0_WD-WCC7K5RYND5D  ONLINE       0     0     0
        ata-WDC_WD40EFAX-68JH4N0_WD-WX61DC9K7J4T  ONLINE       0     0     0

errors: No known data errors

Pool details (zpool get all):

NAME  PROPERTY                       VALUE                          SOURCE
data  size                           14.5T                          -
data  capacity                       58%                            -
data  altroot                        -                              default
data  health                         ONLINE                         -
data  guid                           7433330241577344728            -
data  version                        -                              default
data  bootfs                         -                              default
data  delegation                     on                             default
data  autoreplace                    off                            default
data  cachefile                      -                              default
data  failmode                       wait                           default
data  listsnapshots                  off                            default
data  autoexpand                     off                            default
data  dedupditto                     0                              default
data  dedupratio                     1.00x                          -
data  free                           6.06T                          -
data  allocated                      8.49T                          -
data  readonly                       off                            -
data  ashift                         0                              default
data  comment                        -                              default
data  expandsize                     -                              -
data  freeing                        0                              -
data  fragmentation                  1%                             -
data  leaked                         0                              -
data  multihost                      off                            default
data  checkpoint                     -                              -
data  load_guid                      2011810803420692217            -
data  autotrim                       off                            default
data  feature@async_destroy          enabled                        local
data  feature@empty_bpobj            active                         local
data  feature@lz4_compress           active                         local
data  feature@multi_vdev_crash_dump  enabled                        local
data  feature@spacemap_histogram     active                         local
data  feature@enabled_txg            active                         local
data  feature@hole_birth             active                         local
data  feature@extensible_dataset     active                         local
data  feature@embedded_data          active                         local
data  feature@bookmarks              enabled                        local
data  feature@filesystem_limits      enabled                        local
data  feature@large_blocks           enabled                        local
data  feature@large_dnode            enabled                        local
data  feature@sha512                 enabled                        local
data  feature@skein                  enabled                        local
data  feature@edonr                  enabled                        local
data  feature@userobj_accounting     active                         local
data  feature@encryption             enabled                        local
data  feature@project_quota          active                         local
data  feature@device_removal         enabled                        local
data  feature@obsolete_counts        enabled                        local
data  feature@zpool_checkpoint       enabled                        local
data  feature@spacemap_v2            active                         local
data  feature@allocation_classes     enabled                        local
data  feature@resilver_defer         enabled                        local
data  feature@bookmark_v2            enabled                        local

Pool filesystem details (zfs get all):

NAME  PROPERTY              VALUE                                 SOURCE
data  type                  filesystem                            -
data  creation              Thu Jun 11 12:49 2020                 -
data  used                  6.17T                                 -
data  available             4.07T                                 -
data  referenced            6.17T                                 -
data  compressratio         1.00x                                 -
data  mounted               no                                    -
data  quota                 none                                  default
data  reservation           none                                  default
data  recordsize            128K                                  default
data  mountpoint            /data                                 local
data  sharenfs              off                                   default
data  checksum              on                                    default
data  compression           off                                   default
data  atime                 on                                    default
data  devices               on                                    default
data  exec                  on                                    default
data  setuid                on                                    default
data  readonly              off                                   default
data  zoned                 off                                   default
data  snapdir               hidden                                default
data  aclinherit            restricted                            default
data  createtxg             1                                     -
data  canmount              on                                    default
data  xattr                 on                                    default
data  copies                1                                     default
data  version               5                                     -
data  utf8only              off                                   -
data  normalization         none                                  -
data  casesensitivity       sensitive                             -
data  vscan                 off                                   default
data  nbmand                off                                   default
data  sharesmb              off                                   default
data  refquota              none                                  default
data  refreservation        none                                  default
data  guid                  5544496196900342056                   -
data  primarycache          all                                   default
data  secondarycache        all                                   default
data  usedbysnapshots       0B                                    -
data  usedbydataset         6.17T                                 -
data  usedbychildren        78.4M                                 -
data  usedbyrefreservation  0B                                    -
data  logbias               latency                               default
data  objsetid              54                                    -
data  dedup                 off                                   default
data  mlslabel              none                                  default
data  sync                  standard                              default
data  dnodesize             legacy                                default
data  refcompressratio      1.00x                                 -
data  written               6.17T                                 -
data  logicalused           6.17T                                 -
data  logicalreferenced     6.17T                                 -
data  volmode               default                               default
data  filesystem_limit      none                                  default
data  snapshot_limit        none                                  default
data  filesystem_count      none                                  default
data  snapshot_count        none                                  default
data  snapdev               hidden                                default
data  acltype               posixacl                              local
data  context               none                                  default
data  fscontext             none                                  default
data  defcontext            none                                  default
data  rootcontext           none                                  default
data  relatime              off                                   default
data  redundant_metadata    all                                   default
data  overlay               off                                   default
data  encryption            off                                   default
data  keylocation           none                                  default
data  keyformat             none                                  default
data  pbkdf2iters           0                                     default
data  special_small_blocks  0                                     default
data  omvzfsplugin:uuid     c7a5448c-5449-46af-857e-b9e19a3fb1c7  local

 

[mbosma]

Could you check the output of these commands:

df -h
zfs get written
ls -alh /

Maybe there's a wrong mount/snapshot/symlink that is causing this issue.

 

[scrummie02]

Could you check the output of these commands:

df -h
zfs get written
ls -alh /

Maybe there's a wrong mount/snapshot/symlink that is causing this issue.

df -h

udev            3.8G     0  3.8G   0% /dev
tmpfs           784M   19M  766M   3% /run
/dev/sda1        21G  3.1G   17G  16% /
tmpfs           3.9G   60K  3.9G   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
tmpfs           3.9G   96K  3.9G   1% /tmp

zfs get written

NAME        PROPERTY  VALUE    SOURCE
data        written   6.17T    -
data/Media  written   174K     -

ls -alh

root@omv01:~# ls -alh
total 116K
drwx------  5 root root 4.0K Jun 12 20:24 .
drwxrwxr-x 21 root root 4.0K Jun 11 12:49 ..
-rw-------  1 root root 1.2K Jul  2 08:24 .bash_history
-rw-------  1 root root  446 Mar 25 12:22 .bashrc
drwx------  3 root root 4.0K Jun 11 13:15 .config
-rw-------  1 root root  278 Mar 25 12:22 .inputrc
drwxr-xr-x  3 root root 4.0K Jun 12 20:24 .local
-rw-r--r--  1 root root  70K Jun 11 12:14 openmediavault-omvextrasorg_latest_all5.deb
-rw-r--r--  1 root root  161 Mar 25 12:22 .profile
-rw-r--r--  1 root root   66 Jun 12 20:24 .selected_editor
drwx------  2 root root 4.0K Jun 11 12:03 .ssh
-rw-r--r--  1 root root  215 Jun 18 07:46 .wget-hsts

 

[TRACKER]

ls -alh is showing your user's home directory contents, not root (/) FS contents
execute "ls -alh /" and paste it here

 

[scrummie02]

ls -alh

drwxrwxr-x  21 root root 4.0K Jun 11 12:49 .
drwxrwxr-x  21 root root 4.0K Jun 11 12:49 ..
lrwxrwxrwx   1 root root    7 Jun 11 11:59 bin -> usr/bin
drwxr-xr-x   4 root root 4.0K Jun 11 12:34 boot
drwxr-xr-x  11 root root 4.0K Jul  2 08:22 data
drwxr-xr-x  19 root root 4.4K Jul  2 08:38 dev
drwxrwxr-x  95 root root 4.0K Jul  2 10:27 etc
drwxr-xr-x  11 root root 4.0K Jun 12 15:42 export
drwxr-xr-x   3 root root 4.0K Jun 12 20:10 home
lrwxrwxrwx   1 root root   35 Jun 11 11:59 initrd.img -> boot/initrd.img-5.4.0-0.bpo.4-amd64
lrwxrwxrwx   1 root root   35 Jun 11 11:59 initrd.img.old -> boot/initrd.img-5.4.0-0.bpo.4-amd64
lrwxrwxrwx   1 root root    7 Jun 11 11:59 lib -> usr/lib
lrwxrwxrwx   1 root root    9 Jun 11 11:59 lib32 -> usr/lib32
lrwxrwxrwx   1 root root    9 Jun 11 11:59 lib64 -> usr/lib64
lrwxrwxrwx   1 root root   10 Jun 11 11:59 libx32 -> usr/libx32
drwx------   2 root root  16K Jun 11 11:59 lost+found
drwxr-xr-x   3 root root 4.0K Jun 11 12:00 media
drwxr-xr-x   2 root root 4.0K Mar 25 12:18 mnt
drwxr-xr-x   2 root root 4.0K Mar 25 12:18 opt
dr-xr-xr-x 199 root root    0 Jul  2 08:25 proc
drwx------   5 root root 4.0K Jun 12 20:24 root
drwxr-xr-x  25 root root 1020 Jul  2 11:24 run
lrwxrwxrwx   1 root root    8 Jun 11 11:59 sbin -> usr/sbin
drwxr-xr-x   2 root root 4.0K Mar 17 02:23 sharedfolders
drwxr-xr-x   5 root root 4.0K Jun 11 11:59 srv
dr-xr-xr-x  13 root root    0 Jul  2 08:25 sys
drwxrwxrwt   8 root root  240 Jul  2 11:09 tmp
drwxr-xr-x  13 root root 4.0K Jun 11 12:00 usr
drwxr-xr-x  12 root root 4.0K Jun 11 12:00 var
lrwxrwxrwx   1 root root   32 Jun 11 12:00 vmlinuz -> boot/vmlinuz-5.4.0-0.bpo.4-amd64
lrwxrwxrwx   1 root root   32 Jun 11 12:00 vmlinuz.old -> boot/vmlinuz-5.4.0-0.bpo.4-amd64

 

[EffrafaxOfWug]

Is your ZFS pool meant to be mounted at /data? It doesn't look like it's mounted... what does `zfs mount` show?

 

[scrummie02]

no output, it's blank when I run the ommand

 

[scrummie02]

when running zfs mount -a I get this:
cannot mount '/data': directory is not empty

my folder structure is in /data but nothing is there.

 

[apnar]

Sounds like something caused your ZFS volumes to become unmounted but something else wrote there. I'd try clearing out the /data directory and running the mount command again.

 

[EffrafaxOfWug]

Once a mountpoint is defined, IIRC it's meant to mount automatically but there are things that can stop this, for example if there are files inside /data I think the mount will error out. Worth taking a look in there before you try anything else (`ls -lah /data`).

I'm not really au fait with OMV and only a laymans knowledge of ZFS but the metadata looks correct (mount point is correctly set to /data but mounted==no); assuming it works like standard ZFS you should be able to try to mount it with something like `zfs mount data && zfs mount data/Media`.

 

[scrummie02]

so doing mv /data /data1 and then creating /data and zfs mount -a everyitng works.
Now all my shares are messed up and NFS doesn't work.

 

[apnar]

A quick reboot may help you out with the rest reseting itself.