Amazon S3 compatible ZFS cloud with minIO

gea · Oct 8, 2020

I have put some work into my "Cloud-Filer" concept.
This is an approach to combine a regular filer use of documents that are edited via SMB locally with a Cloud S3 access of same files in a multiuser environment from the Internet.

Problem:
If you allow a cloudsync while files are edited/open or try to open locally while a file is uploaded via S3, a corrupted file may be the result.

Solution:
-Use two filesystems, one for local access, one for S3. Keep needed files or folders in sync (one way or two way based on last edited stamp) via rclone or rsync or other sync tools on demand or based on a timetable and snaps.

-Use ZFS dedup to save datablock only once
- Add a NVMe mirror as a special vdev to hold the dedup table

Problem:
SMB allows authorisation and authentication. You need this on S3 as well based on same users

Solution:
- S3 user/group management that is in sync with SMB user/groups
see MinIO | Learn how to configure MinIO for multiple long term users

I have added this to napp-it 20.dev

Problem
You need policies on S3

I have added policy management:

Problem:
you need to create policies per bucket (none,read,write,readwrite)

Problem:
You need to edit or add these policies later

http://www.napp-it.org/doc/downloads/cloudsync.pdf

Bronko · Mar 6, 2021

More time to play around S3 Cloud Storage on my ZFS Home Filer...

Some Hints:

Have been trying to add a first S3 Cloud >> user and the drop down menu 'to SMB group' under Sync options listed only the build in Local SMB-Groups not my custom ones.

Policy readonly resulted in "Do not have enough permissions to access this resource" when I access the bucket either for user or group or both (no objects in bucket visable); login works.

gea · Mar 8, 2021

Hello Bronko
i have fixed the problem of missing SMB groups in 21.dev

In general the user management of minio is quite new.
In my tests last year I have also seen a policy behaviour that was not always fully predictable or fully understandable for me (I have played only with some basic policy settings as there are many policy options that you may need to combine). Some settings allow a write/update ex via rclone but not a bucket list in the webbrowser.

Consider policies as a "as is" state to play with and update minio from time to time to newest.
For more details to play with see MinIO | Learn how to configure MinIO for multiple long term users

Bronko · Mar 8, 2021

Whenever possible I'm willing to test your napp-it implementation to push it to something better for all of us.

Thanks a lot for your hints, will stay tuned and playing currently with HAProxy in front of MinIO for https offloading by certificate management in a single place (firewall) and not on each 'backend' server. And it works... ;-)

gea said:
i have fixed the problem of missing SMB groups in 21.dev

confirmed.

gea · Mar 8, 2021

Current State is:
Minio/s3 bucket policy managent is not at a level where Windows ntfs and Solarish SMB is for years. But for many use cases this is ok

Bronko · Mar 9, 2021

Regarding readonly Policy issue I haved added "s3:ListBucket" Action in to the napp-it Policy File:

>>User >>Appliance S3 Cloud >>Policy >>edit readonly:

JSON:

{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Effect": "Allow",
         "Action": ["s3:GetObject","s3:ListBucket","s3:GetBucketLocation"],
         "Resource": ["arn:aws:s3:::*"
         ]
      }
   ]
}

Now it seems to work...

bugmenot · Apr 19, 2021

Hi Gea,

first of: Thanks for all the work you are putting into this.

I was testing v.19.2, 20.dev (which wasn't possible to update to from the GUI?) and 21.dev for with 151034e from the .ova and also updated to 151036 (newest? v? if I recall correctly) for the Minio integration and ran into problems. I believe the select code for the instance in actions.pl for S3 users, group, and policy to be faulty. If you use the default cert location (path contains minio) or call your dataset e.g minio etc. - it doesn't seem to work:

Perl:

#print "Select a minIO instance<hr>";
    $t=`ps axw | grep minio | grep -v grep | grep server`; chomp ($t);
    if ($t eq "") { $t="none"; }

    @t=split(/\n/,$t);
    foreach my $l (@t) {
       #$l=~s/.*minio/minio/;  #greedy problem
       $l=~s/.*minio server/minio server/; #quickfix for now
    }
    $t=join(",",@t);

Now potentially, paths and options with "minio server" also become problematic

.

A better documentation of how to specify options with

Alt minio server

would be helpful and highly appriciated.

Another problem I encountered on 151036 was, getting some error message along the line of

"Invalid" "JSON" "RPC" "response" "UI?" "newer?" "version?" "does not match?"

(I don't recall it 100% sorry) while trying to log into the Minio Browser Webinterface.
I tried different browsers and It wouldn't let me in.

Maybe related to:

Important OmniOS Security Update r36m, r34am, r30cm - Security / Features/ Bugfixes
Support for secure RPC for the Netlogon protocol between OmniOS systems and Microsoft Active Directory servers is added to all OmniOS versions under support. This is required to fully mitigate CVE-2020-1472, and will shortly be enforced by Windows domain controllers.
If you use Windows Active Directory you should at least evaluate.

I didn't dig far into it however.
Hope that helps anyway.

gea · Apr 20, 2021

I will check that for next release.
It seems that the minIO Browser requires now https while current server default is http. For https you need to place a public.cert and a private.key in /var/web-gui/_log/minio but I suppose some more tests are needed for current version as the error message remains even with a certificate there.

In the meantime you can go back to the last minIO binary that is in 151032

Code:

pkg uninstall minio
pkg uninstall minio-mc

pkg unset-publisher extra.omnios
pkg set-publisher -g https://pkg.omniosce.org/r151032/extra extra.omnios

pkg install minio
pkg install minio-mc

about alt server setting in the S3 share menu
when you set altserver=xx there, the start for minio will like
/opt/ooce/minio/bin/minio server xx

the default is ex
/opt/ooce/minio/bin/minio server server /data/xx/S3_data --address :9000

btw
In above example (share data/xx), you will find a
/data/xx/S3_config/minio.sh

This is the minio server startup command.
When disabled, you can start minio manually with this shell script for tests from console, ex
sh /data/xx/S3_config/minio.sh

bugmenot · Apr 20, 2021

Thanks.
I searched github and found the error message.
As you may know by now, it reads:
"Invalid UI version in the JSON-RPC response".
The corresponding code can be found at:

minio/browser/app/js/web.js at 48c5e7e5b62932c2702dcfcb2c58c582cab0ed0f · minio/minio

MinIO is a high-performance, S3 compatible object store, open sourced under GNU AGPLv3 license. - minio/minio

github.com

On first look, it seems as if in this particular case it may be just a version mismatch of some kind, that's causing it. Maybe there is something deeper at play after all - One will have to take a closer look.

gea · Apr 21, 2021

I have created a request at omnios-discuss with some more details

Topicbox

bugmenot · Apr 21, 2021

Good.
It looks to me like a general minio problem though ...
See 782037 – net-fs/minio-2021.04.06.23.11.00: webapp "Invalid UI version in the JSON-RPC response" on login
Also from that I took a closer look at:

Comparing RELEASE.2021-03-04T00-53-13Z...RELEASE.2021-03-10T05-11-33Z · minio/minio

MinIO is a high-performance, S3 compatible object store, open sourced under GNU AGPLv3 license. - Comparing RELEASE.2021-03-04T00-53-13Z...RELEASE.2021-03-10T05-11-33Z · minio/minio

github.com

file: cmd/web-handlers.go
- dependency of "github.com/minio/minio/browser"
- reply.UIVersion = browser.UIVersion
+ reply.UIVersion = Version
May be near the root of the problem.

gea · Apr 21, 2021

Some hours later, fixed by Andy Fiddaman, OmniOS

Topicbox

Bronko · Nov 7, 2021

Bronko said:
Thanks a lot for your hints, will stay tuned and playing currently with HAProxy in front of MinIO for https offloading by certificate management in a single place (firewall) and not on each 'backend' server. And it works... ;-)

Hi @gea ,
have been updated my home system from omnios-r151034 -> omnios-r151038.
It seems starting now or before minio Console endpoint is listening on a dynamic port and my HAProxy failed to work from outside.

To be up and running again without investigation for a new HAProxy configuration

I tried to understand your 'Alt minio server' section in creating server instance mask at ZFS-Filesystem menu.

What I found is I have not only to add my additional settings ( --console-address :9001 to choose a static port for endpoint) but server address and data directory too.

Alt minio server:
/tank1/Data/shareS3/S3_data --address :9000 --console-address :9001
(--console-address cannot be same as --address)

These settings are gone by re enabling the same server instance (ZFS-Filesystem), You have always to add again.
But it works as expected.

Btw. I found these at console by manual testing:

Code:

WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD

gea · Nov 9, 2021

Update: napp-it 20.dev from nov 09 supports the new minIO settings (required for minIO newer may 2021)

- ROOT_USER and ROOT_PASSWORD instead the former KEY and SECRET
- a new webconsole at port 800x (1000 lower than service port) with support for user, groups and permissions
- /var/web-gui/_log/minio/certs is new cert directory

Bronko · Nov 9, 2021

Regarding this point new version works as expected, but my s3user and s3groups are gone, but no problem.
Certs folder doesn't tested due to HAProxy in front of.

New issue:
++ group entry in Grouplist is missing, but new group at new user creation is auto created.

It seems all policies back on default, have to change readonly policy as described here to List Bucket again.

gea · Nov 10, 2021

minIO has changed a lot up from may 2021. Settings mostly moved from local to the backend and user, group and permission management is done via the new minIO console.

The according napp-it settings will need to be removed or need a whole rework

BobTB · Oct 19, 2023

Is this reworked in the latest napp-it? Does it actually work as expected with the newest minIO?

gea · Oct 20, 2023

When I last checked, minIO worked with its improved user management and port access/management. Just enable S3 sharing in menu ZFS filesystems with an admin user+pw. Then connect S3 at the given port with that user/pw. This is the admin user that can manage S3. The napp-it S3 user menu is mostly obsolete unless you want to check S3 policies within napp-it. It is removed on next release.

Bronko · Nov 6, 2023

gea said:
The napp-it S3 user menu is mostly obsolete...

But we will loose these hole idea from above...

gea said:
Solution:
- S3 user/group management that is in sync with SMB user/groups

gea · Nov 6, 2023

When I wrote this module, minIO lacked user and group management.
Now i hesitate to include this as a substition of minIO functions as there are no further tests of correct functionality on newer minIO versions.

You can use the menu in current napp-it 23.06. You can copy the menufolder to newer releases or use the menu as a private menu (copy menu folder "/var/web-gui/data_23.06/napp-it/zfsos/04_User and user-groups=-lin/15_Appliance S3 Cloud/" to /var/web-gui/_my/zfsos/. This is update save.

You can use it as long as it works but without support or create users and groups on minIO directly but must sync names and passwords manually.

Amazon S3 compatible ZFS cloud with minIO

Well-Known Member

Member

Well-Known Member

Member

Well-Known Member

Member

Member

Well-Known Member

Member

Well-Known Member

Member

Well-Known Member

Member

Well-Known Member

Member

Well-Known Member

Member

Well-Known Member

Member

Well-Known Member