MergerFS mounted folder no permission on Ubuntu 22.04

[Bert]

I am combining multiple disks via merger.fs. This used to work without any issues on Debian. Recently, I need to switch to Ubuntu 22.04. I am having this problem when I tried to use merger.fs on ubuntu.

- If I started merger.fs via mount -a from /etc/fstab, no one including root can access to the mergerfs folder. There is no list, read or write access to the mount point.
- If I started merger.fs via command line by root/sudo, no one except root can access to the mergerfs folder.

I am able to access to the mount only by restarting the server when the mount is declared in fstab. Then everyone can access but obvious problem here is that I need to restart the server.

Does anyone know what the problem here is? Is there a work around?

PS:
There seems to be bunch of security model changes in ubuntu over debian. I see strange permission issues with xrdp as well.

 

[rubylaser]

Hello, I'm running mergerfs on Ubuntu 22.04 without issue. This definitely sounds like permissions issue. Could you please share the contents of your /etc/fstab and the permissions of each of your disks' mountpoints. If your disks are mount at /disks/data you'd do the following.

sudo -i
cat /etc/fstab
ls -la /disks/data

Are you install mergerfs from dpkg from Trapexit's Github page?

mergerfs -V

That should give some info to start diagnosing the problem.

 

[Bert]

I installed mergerfs from standard distro via apt install mergerFS. I will get the rest of the info tonight.

 

[Bert]

@rubylaser , here is a sample entry from my fstab:
UUID=7d884cc1-5c05-476d-9326-9edf10580ddf /chia/nft/mnt99 xfs defaults,nofail 0 2

here is mergerfs entry
#/nft/* /pool fuse.mergerfs nonempty,direct_io,defaults,allow_other,minfreespace=3G,moveonenospc=true,category.create=lfs,fsname=mergerfsPool 0 0

I did ls -la on the drives and here is how it looks like (too many to list):

Context – share whatever you see with others in seconds

ctxt.io

My mergerfs version is:
mergerfs version: 2.33.3

 

[rubylaser]

Thanks! That's some good info to get started. After reviewing your message, this doesn't look like a permissions error since the disks have mix of 775 and 777 permissions. So, the first thing you will want to do is update your version of mergerfs. The one in the repository is very old (came out in December 2021) and instead install from Trapexit's Github (he's the developer of mergerfs).

Let's remove the version from the Ubuntu repo first. I'm running all of these commands as the root user.

sudo -i
umount /pool
apt remove mergerfs

Then, grab the latest version of mergerfs. This assumes that you are on x86 and running a 64 bit kernel. I keep these directions up to date on my website if you'd like to refer to them in the future.

wget https://github.com/trapexit/mergerfs/releases/download/2.35.1/mergerfs_2.35.1.ubuntu-jammy_amd64.deb
dpkg -i mergerfs_2.35.1.ubuntu-jammy_amd64.deb
mergerfs -V
# should return 2.35.1
rm mergerfs*.deb

Now, that you are running a recent version of mergerfs, there are some things you need to do to your /etc/fstab. You are using a couple of deprecated mount options and your mergerfs mount line is pointing at the wrong directory (unless it was a copy/paste error).

So, your new mergerfs entry should look like the below. This is assuming that all of your disks are mounted in /chia/nft as you've shown for your example disk. Your previous mergerfs mount was collecting disks mounted at /nft, the wrong directory, rather than /chia/nft as you showed in your example disk, so I've edited that as well.

/chia/nft/* /pool fuse.mergerfs cache.files=auto-full,dropcacheonclose=true,category.create=lfs,moveonenospc=true,minfreespace=3G,fsname=mergerfsPool,nonempty 0 0

mount -a

Also, make certain that your /pool directory has 775 or 777 permissions to match your disks. Another question from me. Do the permissions really need to be that loose (775/777)? I'd think that 755 would be more than enough.

Finally, you can always run mount -a as the root user when the system is running to re-run your mounts in /etc/fstab without needing to reboot the system.

Please let me know if this makes sense.

 

[Bert]

Thank you very much for your detailed explanation.

I made a mistake while copy pasting the mergers directive. The directive is actually correct and it is same as you shared with me. As I mentioned mergerfs works as expected if I reboot the server.

This problem with using mount - a is that it removes all access permissions on the mergerfs mount for usets. I cannot change the permissions on it with chmod or chown. That's why I was asking for help.

It seems like next action item for me is to update the mergerfs version. I will do that over the weekend. I

 

[rubylaser]

Thank you very much for your detailed explanation.

I made a mistake while copy pasting the mergers directive. The directive is actually correct and it is same as you shared with me. As I mentioned mergerfs works as expected if I reboot the server.

This problem with using mount - a is that it removes all access permissions on the mergerfs mount for usets. I cannot change the permissions on it with chmod or chown. That's why I was asking for help.

It seems like next action item for me is to update the mergerfs version. I will do that over the weekend. I

I'm having a difficult time fully understanding your issue, so I'm just going to ask a few more questions. mount -a doesn't change permissions, it literally just mounts the directories you have in /etc/fstab. If you unmount your mergerfs pool (umount /pool). What are the permissions of that empty folder (ls -la /pool)?

Are you making sure to run any chown/chmod commands as the root user or with sudo? You should absolutely be able to change permissions on the pool's mountpoint, on the individual disks, and any files or directories, even when it's mounted via mergerfs if you are using a user with permssions to do those actions.

Do you have your mergerfs line in /etc/fstab at the bottom of the file (after all of the disks are mounted)?

Hopefully, stepping up to the newer version of mergerfs will help resolve some of these issues.

 

[Bert]

I admit the problem is really weird. I will share ls -la results for the mount points in detail over the weekend.

 

[rubylaser]

Any update?

 

[Bert]

Thanks for the follow up @rubylaser. Since that time, I switched my production unit to debian.

Though after my restart, I was not able to repro the issue on the ubuntu system. My guess is that updating the mergers package to the latest version has fixed the issue. Thank you for your help.