Amazon S3 compatible ZFS cloud with minIO

gea

Well-Known Member
Dec 31, 2010
2,649
908
113
DE
I have put some work into my "Cloud-Filer" concept.
This is an approach to combine a regular filer use of documents that are edited via SMB locally with a Cloud S3 access of same files in a multiuser environment from the Internet.

Problem:
If you allow a cloudsync while files are edited/open or try to open locally while a file is uploaded via S3, a corrupted file may be the result.

Solution:
-Use two filesystems, one for local access, one for S3. Keep needed files or folders in sync (one way or two way based on last edited stamp) via rclone or rsync or other sync tools on demand or based on a timetable and snaps.

-Use ZFS dedup to save datablock only once
- Add a NVMe mirror as a special vdev to hold the dedup table

Problem:
SMB allows authorisation and authentication. You need this on S3 as well based on same users

Solution:
- S3 user/group management that is in sync with SMB user/groups
see MinIO | Learn how to configure MinIO for multiple long term users

I have added this to napp-it 20.dev

s3user.png

Problem
You need policies on S3

I have added policy management:
policies.png

Problem:
you need to create policies per bucket (none,read,write,readwrite)

create_policy.png

Problem:
You need to edit or add these policies later
edit.png

 
Last edited:

Bronko

Member
May 13, 2016
106
9
18
102
More time to play around S3 Cloud Storage on my ZFS Home Filer...

Some Hints:

Have been trying to add a first S3 Cloud >> user and the drop down menu 'to SMB group' under Sync options listed only the build in Local SMB-Groups not my custom ones.

Policy readonly resulted in "Do not have enough permissions to access this resource" when I access the bucket either for user or group or both (no objects in bucket visable); login works.
 
Last edited:

gea

Well-Known Member
Dec 31, 2010
2,649
908
113
DE
Hello Bronko
i have fixed the problem of missing SMB groups in 21.dev

In general the user management of minio is quite new.
In my tests last year I have also seen a policy behaviour that was not always fully predictable or fully understandable for me (I have played only with some basic policy settings as there are many policy options that you may need to combine). Some settings allow a write/update ex via rclone but not a bucket list in the webbrowser.

Consider policies as a "as is" state to play with and update minio from time to time to newest.
For more details to play with see MinIO | Learn how to configure MinIO for multiple long term users
 
Last edited:

Bronko

Member
May 13, 2016
106
9
18
102
Whenever possible I'm willing to test your napp-it implementation to push it to something better for all of us.

Thanks a lot for your hints, will stay tuned and playing currently with HAProxy in front of MinIO for https offloading by certificate management in a single place (firewall) and not on each 'backend' server. And it works... ;-)

i have fixed the problem of missing SMB groups in 21.dev
confirmed.
 
Last edited:

gea

Well-Known Member
Dec 31, 2010
2,649
908
113
DE
Current State is:
Minio/s3 bucket policy managent is not at a level where Windows ntfs and Solarish SMB is for years. But for many use cases this is ok
 

Bronko

Member
May 13, 2016
106
9
18
102
Regarding readonly Policy issue I haved added "s3:ListBucket" Action in to the napp-it Policy File:

>>User >>Appliance S3 Cloud >>Policy >>edit readonly:

JSON:
{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Effect": "Allow",
         "Action": ["s3:GetObject","s3:ListBucket","s3:GetBucketLocation"],
         "Resource": ["arn:aws:s3:::*"
         ]
      }
   ]
}

Now it seems to work...
 
  • Like
Reactions: gea

bugmenot

New Member
Aug 14, 2017
5
0
1
Hi Gea,

first of: Thanks for all the work you are putting into this.

I was testing v.19.2, 20.dev (which wasn't possible to update to from the GUI?) and 21.dev for with 151034e from the .ova and also updated to 151036 (newest? v? if I recall correctly) for the Minio integration and ran into problems. I believe the select code for the instance in actions.pl for S3 users, group, and policy to be faulty. If you use the default cert location (path contains minio) or call your dataset e.g minio etc. - it doesn't seem to work:

Perl:
#print "Select a minIO instance<hr>";
    $t=`ps axw | grep minio | grep -v grep | grep server`; chomp ($t);
    if ($t eq "") { $t="none"; }

    @t=split(/\n/,$t);
    foreach my $l (@t) {
       #$l=~s/.*minio/minio/;  #greedy problem
       $l=~s/.*minio server/minio server/; #quickfix for now
    }
    $t=join(",",@t);
Now potentially, paths and options with "minio server" also become problematic ;).

A better documentation of how to specify options with
Alt minio server
would be helpful and highly appriciated.

Another problem I encountered on 151036 was, getting some error message along the line of
"Invalid" "JSON" "RPC" "response" "UI?" "newer?" "version?" "does not match?"
(I don't recall it 100% sorry) while trying to log into the Minio Browser Webinterface.
I tried different browsers and It wouldn't let me in.

Maybe related to:
Important OmniOS Security Update r36m, r34am, r30cm - Security / Features/ Bugfixes
Support for secure RPC for the Netlogon protocol between OmniOS systems and Microsoft Active Directory servers is added to all OmniOS versions under support. This is required to fully mitigate CVE-2020-1472, and will shortly be enforced by Windows domain controllers.
If you use Windows Active Directory you should at least evaluate.
I didn't dig far into it however.
Hope that helps anyway.
 

gea

Well-Known Member
Dec 31, 2010
2,649
908
113
DE
I will check that for next release.
It seems that the minIO Browser requires now https while current server default is http. For https you need to place a public.cert and a private.key in /var/web-gui/_log/minio but I suppose some more tests are needed for current version as the error message remains even with a certificate there.

In the meantime you can go back to the last minIO binary that is in 151032

Code:
pkg uninstall minio
pkg uninstall minio-mc

pkg unset-publisher extra.omnios
pkg set-publisher -g https://pkg.omniosce.org/r151032/extra extra.omnios

pkg install minio
pkg install minio-mc

about alt server setting in the S3 share menu
when you set altserver=xx there, the start for minio will like
/opt/ooce/minio/bin/minio server xx

the default is ex
/opt/ooce/minio/bin/minio server server /data/xx/S3_data --address :9000

btw
In above example (share data/xx), you will find a
/data/xx/S3_config/minio.sh

This is the minio server startup command.
When disabled, you can start minio manually with this shell script for tests from console, ex
sh /data/xx/S3_config/minio.sh
 
Last edited:

bugmenot

New Member
Aug 14, 2017
5
0
1
Thanks.
I searched github and found the error message.
As you may know by now, it reads:
"Invalid UI version in the JSON-RPC response".
The corresponding code can be found at:
On first look, it seems as if in this particular case it may be just a version mismatch of some kind, that's causing it. Maybe there is something deeper at play after all - One will have to take a closer look.
 

bugmenot

New Member
Aug 14, 2017
5
0
1