Amazon S3 compatible ZFS cloud with minIO

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

gea

Well-Known Member
Dec 31, 2010
3,141
1,182
113
DE
I have put some work into my "Cloud-Filer" concept.
This is an approach to combine a regular filer use of documents that are edited via SMB locally with a Cloud S3 access of same files in a multiuser environment from the Internet.

Problem:
If you allow a cloudsync while files are edited/open or try to open locally while a file is uploaded via S3, a corrupted file may be the result.

Solution:
-Use two filesystems, one for local access, one for S3. Keep needed files or folders in sync (one way or two way based on last edited stamp) via rclone or rsync or other sync tools on demand or based on a timetable and snaps.

-Use ZFS dedup to save datablock only once
- Add a NVMe mirror as a special vdev to hold the dedup table

Problem:
SMB allows authorisation and authentication. You need this on S3 as well based on same users

Solution:
- S3 user/group management that is in sync with SMB user/groups
see MinIO | Learn how to configure MinIO for multiple long term users

I have added this to napp-it 20.dev

s3user.png

Problem
You need policies on S3

I have added policy management:
policies.png

Problem:
you need to create policies per bucket (none,read,write,readwrite)

create_policy.png

Problem:
You need to edit or add these policies later
edit.png

 
Last edited:

Bronko

Member
May 13, 2016
111
10
18
105
More time to play around S3 Cloud Storage on my ZFS Home Filer...

Some Hints:

Have been trying to add a first S3 Cloud >> user and the drop down menu 'to SMB group' under Sync options listed only the build in Local SMB-Groups not my custom ones.

Policy readonly resulted in "Do not have enough permissions to access this resource" when I access the bucket either for user or group or both (no objects in bucket visable); login works.
 
Last edited:

gea

Well-Known Member
Dec 31, 2010
3,141
1,182
113
DE
Hello Bronko
i have fixed the problem of missing SMB groups in 21.dev

In general the user management of minio is quite new.
In my tests last year I have also seen a policy behaviour that was not always fully predictable or fully understandable for me (I have played only with some basic policy settings as there are many policy options that you may need to combine). Some settings allow a write/update ex via rclone but not a bucket list in the webbrowser.

Consider policies as a "as is" state to play with and update minio from time to time to newest.
For more details to play with see MinIO | Learn how to configure MinIO for multiple long term users
 
Last edited:

Bronko

Member
May 13, 2016
111
10
18
105
Whenever possible I'm willing to test your napp-it implementation to push it to something better for all of us.

Thanks a lot for your hints, will stay tuned and playing currently with HAProxy in front of MinIO for https offloading by certificate management in a single place (firewall) and not on each 'backend' server. And it works... ;-)

i have fixed the problem of missing SMB groups in 21.dev
confirmed.
 
Last edited:

gea

Well-Known Member
Dec 31, 2010
3,141
1,182
113
DE
Current State is:
Minio/s3 bucket policy managent is not at a level where Windows ntfs and Solarish SMB is for years. But for many use cases this is ok
 

Bronko

Member
May 13, 2016
111
10
18
105
Regarding readonly Policy issue I haved added "s3:ListBucket" Action in to the napp-it Policy File:

>>User >>Appliance S3 Cloud >>Policy >>edit readonly:

JSON:
{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Effect": "Allow",
         "Action": ["s3:GetObject","s3:ListBucket","s3:GetBucketLocation"],
         "Resource": ["arn:aws:s3:::*"
         ]
      }
   ]
}

Now it seems to work...
 
  • Like
Reactions: gea

bugmenot

Member
Aug 14, 2017
56
17
8
Hi Gea,

first of: Thanks for all the work you are putting into this.

I was testing v.19.2, 20.dev (which wasn't possible to update to from the GUI?) and 21.dev for with 151034e from the .ova and also updated to 151036 (newest? v? if I recall correctly) for the Minio integration and ran into problems. I believe the select code for the instance in actions.pl for S3 users, group, and policy to be faulty. If you use the default cert location (path contains minio) or call your dataset e.g minio etc. - it doesn't seem to work:

Perl:
#print "Select a minIO instance<hr>";
    $t=`ps axw | grep minio | grep -v grep | grep server`; chomp ($t);
    if ($t eq "") { $t="none"; }

    @t=split(/\n/,$t);
    foreach my $l (@t) {
       #$l=~s/.*minio/minio/;  #greedy problem
       $l=~s/.*minio server/minio server/; #quickfix for now
    }
    $t=join(",",@t);
Now potentially, paths and options with "minio server" also become problematic ;).

A better documentation of how to specify options with
Alt minio server
would be helpful and highly appriciated.

Another problem I encountered on 151036 was, getting some error message along the line of
"Invalid" "JSON" "RPC" "response" "UI?" "newer?" "version?" "does not match?"
(I don't recall it 100% sorry) while trying to log into the Minio Browser Webinterface.
I tried different browsers and It wouldn't let me in.

Maybe related to:
Important OmniOS Security Update r36m, r34am, r30cm - Security / Features/ Bugfixes
Support for secure RPC for the Netlogon protocol between OmniOS systems and Microsoft Active Directory servers is added to all OmniOS versions under support. This is required to fully mitigate CVE-2020-1472, and will shortly be enforced by Windows domain controllers.
If you use Windows Active Directory you should at least evaluate.
I didn't dig far into it however.
Hope that helps anyway.
 

gea

Well-Known Member
Dec 31, 2010
3,141
1,182
113
DE
I will check that for next release.
It seems that the minIO Browser requires now https while current server default is http. For https you need to place a public.cert and a private.key in /var/web-gui/_log/minio but I suppose some more tests are needed for current version as the error message remains even with a certificate there.

In the meantime you can go back to the last minIO binary that is in 151032

Code:
pkg uninstall minio
pkg uninstall minio-mc

pkg unset-publisher extra.omnios
pkg set-publisher -g https://pkg.omniosce.org/r151032/extra extra.omnios

pkg install minio
pkg install minio-mc

about alt server setting in the S3 share menu
when you set altserver=xx there, the start for minio will like
/opt/ooce/minio/bin/minio server xx

the default is ex
/opt/ooce/minio/bin/minio server server /data/xx/S3_data --address :9000

btw
In above example (share data/xx), you will find a
/data/xx/S3_config/minio.sh

This is the minio server startup command.
When disabled, you can start minio manually with this shell script for tests from console, ex
sh /data/xx/S3_config/minio.sh
 
Last edited:

bugmenot

Member
Aug 14, 2017
56
17
8
Thanks.
I searched github and found the error message.
As you may know by now, it reads:
"Invalid UI version in the JSON-RPC response".
The corresponding code can be found at:
On first look, it seems as if in this particular case it may be just a version mismatch of some kind, that's causing it. Maybe there is something deeper at play after all - One will have to take a closer look.
 

bugmenot

Member
Aug 14, 2017
56
17
8
Good.
It looks to me like a general minio problem though ...
See 782037 – net-fs/minio-2021.04.06.23.11.00: webapp "Invalid UI version in the JSON-RPC response" on login
Also from that I took a closer look at:
file: cmd/web-handlers.go
- dependency of "github.com/minio/minio/browser"
- reply.UIVersion = browser.UIVersion
+ reply.UIVersion = Version
May be near the root of the problem.
 

Bronko

Member
May 13, 2016
111
10
18
105
Thanks a lot for your hints, will stay tuned and playing currently with HAProxy in front of MinIO for https offloading by certificate management in a single place (firewall) and not on each 'backend' server. And it works... ;-)
Hi @gea ,
have been updated my home system from omnios-r151034 -> omnios-r151038.
It seems starting now or before minio Console endpoint is listening on a dynamic port and my HAProxy failed to work from outside.

To be up and running again without investigation for a new HAProxy configuration ;) I tried to understand your 'Alt minio server' section in creating server instance mask at ZFS-Filesystem menu.

What I found is I have not only to add my additional settings ( --console-address :9001 to choose a static port for endpoint) but server address and data directory too.

Alt minio server:
/tank1/Data/shareS3/S3_data --address :9000 --console-address :9001
(--console-address cannot be same as --address)

These settings are gone by re enabling the same server instance (ZFS-Filesystem), You have always to add again.
But it works as expected.

Btw. I found these at console by manual testing:

Code:
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
 
Last edited:
  • Like
Reactions: gea

gea

Well-Known Member
Dec 31, 2010
3,141
1,182
113
DE
Update: napp-it 20.dev from nov 09 supports the new minIO settings (required for minIO newer may 2021)

- ROOT_USER and ROOT_PASSWORD instead the former KEY and SECRET
- a new webconsole at port 800x (1000 lower than service port) with support for user, groups and permissions
- /var/web-gui/_log/minio/certs is new cert directory
 
Last edited:
  • Like
Reactions: Bronko

Bronko

Member
May 13, 2016
111
10
18
105
Regarding this point new version works as expected, but my s3user and s3groups are gone, but no problem.
Certs folder doesn't tested due to HAProxy in front of.

New issue:
++ group entry in Grouplist is missing, but new group at new user creation is auto created.

It seems all policies back on default, have to change readonly policy as described here to List Bucket again.
 

gea

Well-Known Member
Dec 31, 2010
3,141
1,182
113
DE
minIO has changed a lot up from may 2021. Settings mostly moved from local to the backend and user, group and permission management is done via the new minIO console.

The according napp-it settings will need to be removed or need a whole rework
 

BobTB

Member
Jul 19, 2019
81
19
8
Is this reworked in the latest napp-it? Does it actually work as expected with the newest minIO?
 

gea

Well-Known Member
Dec 31, 2010
3,141
1,182
113
DE
When I last checked, minIO worked with its improved user management and port access/management. Just enable S3 sharing in menu ZFS filesystems with an admin user+pw. Then connect S3 at the given port with that user/pw. This is the admin user that can manage S3. The napp-it S3 user menu is mostly obsolete unless you want to check S3 policies within napp-it. It is removed on next release.

s3.PNG
 
Last edited:
  • Like
Reactions: Bronko and gb00s

gea

Well-Known Member
Dec 31, 2010
3,141
1,182
113
DE
When I wrote this module, minIO lacked user and group management.
Now i hesitate to include this as a substition of minIO functions as there are no further tests of correct functionality on newer minIO versions.

You can use the menu in current napp-it 23.06. You can copy the menufolder to newer releases or use the menu as a private menu (copy menu folder "/var/web-gui/data_23.06/napp-it/zfsos/04_User and user-groups=-lin/15_Appliance S3 Cloud/" to /var/web-gui/_my/zfsos/. This is update save.

You can use it as long as it works but without support or create users and groups on minIO directly but must sync names and passwords manually.
 
  • Like
Reactions: Bronko and gb00s