Amazon S3 compatible ZFS cloud with minIO

[gea]

Due a customer/my request, OmniOS added minIO client and server and rclone
to the extra repo Package Catalog

MinIO | Enterprise Grade, High Performance Object Storage

MinIO | MinIO Client Quickstart Guide
MinIO | The MinIO Quickstart Guide

MinIO Enjoying Role in Emerging Cloud Architecture

As backup/sharing client you can use

- your browser, via webclient and http://ip::9000, part of minIO
- rclone see rclone - rsync for cloud storage
- Duplicati, a Windows, OSX, Linux backup application , see Duplicati
- Veeam, see Adding S3 Compatible Object Storage - Veeam Backup Guide for vSphere

- Other sync tools (google: s3 sync tools) ex
5 Best Amazon S3 User interface tools | GUIs for Amazon simple storage

Install minIO (current OmniOS)

pkg set-publisher -g https://pkg.omniosce.org/r151032/extra extra.omnios

pkg install minio
pkg install minio-mc
pkg install rclone

#optionally enable minio as a service
# (I prefer a manual start with options, you can start multiple instances with a different port)
svcadm enable minio

# service manifest, see /lib/svc/manifest/application/application-minio.xml

Start minIO manually

1.
#Create datadir ex /s3/data and configdir /s3/config

2.
#Start minIO (credidentials must be set via export)
export MINIO_ACCESS_KEY=minio;
export MINIO_SECRET_KEY=minio1234;
/opt/ooce/minio/bin/minio server /s3/data --config-dir /s3/config;

3.
#Start Browser ex Google Crome to view, delete, upload, download data via browser
http://ip:9000

enter the above credidentials (ex minio and minio1234)

If you want to modify accessdata:
This requires to export old and new values ex:

export MINIO_ACCESS_KEY=mini2;
export MINIO_SECRET_KEY=minio134;
export MINIO_ACCESS_KEY_OLD=minio;
export MINIO_SECRET_KEY_OLD=minio133;
/opt/ooce/minio/bin/minio server /s3/data --config-dir /s3/config;

Start options of miniIO

USAGE:
  minio [FLAGS] COMMAND [ARGS...]

COMMANDS:
  server   start object storage server
  gateway  start object storage gateway

FLAGS:
  --config-dir value, -C value  [DEPRECATED] path to legacy configuration directory (default: "/root/.minio")
  --address value               bind to a specific ADDRESS:PORT, ADDRESS can be an IP or hostname (default: ":9000")
  --certs-dir value, -S value   path to certs directory (default: "/root/.minio/certs")
  --quiet                       disable startup information
  --anonymous                   hide sensitive information from logging
  --json                        output server logs and startup information in json format
  --compat                      enable strict S3 compatibility by turning off certain performance optimizations
  --help, -h                    show help
  --version, -v                 print the version

Windows and TLS
see Windows service + minio with ssl error


Optional: start minIO in distributed/ clustered mode

output from:  ./minio server

EXAMPLES:
  1. Start minio server on "/home/shared" directory.
     $ minio server /home/shared

  2. Start distributed minio server on an 32 node setup with 32 drives each, run following command on all the nodes
     $ export MINIO_ACCESS_KEY=minio
     $ export MINIO_SECRET_KEY=miniostorage
     $ minio server http://node{1...32}.example.com/mnt/export/{1...32}

  3. Start distributed minio server in an expanded setup, run the following command on all the nodes
     $ export MINIO_ACCESS_KEY=minio
     $ export MINIO_SECRET_KEY=miniostorage
     $ minio server http://node{1...16}.example.com/mnt/export/{1...32} \
            http://node{17...64}.example.com/mnt/export/{1...64}

 

[Rand__]

Ok, just for my understanding - this allows me to run a S3 compatible storage at home on my Napp-It box?
So whenever anything touts S3 integration I can run that against my own box?

That would be effing brilliant for all those stupid (to me) cloud (only) integrated apps nowadays.

 

[gea]

Yes, this is the point, an inhouse replacement of S3 as an addition to a ZFS filer.
OmniOS added minIO this week due a request for an S3 compatible backup destination for a Veeam environment. This makes a huge difference over compiling yourself.

I have not tried minIO myself, this will be the next step after encryption is integrated.
As far as I see, all what is needed is a basic Howto, a setup script (wget..) that installs miniIO, downloads rclone and does a basic setup. Within napp-it propably a menu for service management, configuration and optionally Cluster management if more than one miniIO server is intended. Just to make it as easy as the ZFS filer.

In a second step I may include it then in a basic napp-it setup as a default feature.

 

[StevenDTX]

Ok, just for my understanding - this allows me to run a S3 compatible storage at home on my Napp-It box?
So whenever anything touts S3 integration I can run that against my own box?

That would be effing brilliant for all those stupid (to me) cloud (only) integrated apps nowadays.

I have used Minio for years. I installed Duplicati on all of my family's computers and they all back up to me.

 

[Rand__]

That sounds great

Anything that can do Dropbox too?

 

[gregsachs]

I have used Minio for years. I installed Duplicati on all of my family's computers and they all back up to me.

Ditto, I run my offsite backup to a Pi with Minio and USB disk.

 

[gea]

I have got an info from OmniOS as they now added `rclone` as
well to the -extra repositories for releases r30/r32/bloody.

A remark to the minio server package:
it ships with a default instance using /var/opt/ooce/minio as its data
root (in accordance to their -extra packages layout).
however new SMF instances can be created and the SMF `datadir` be set to
a different dataset/path. (A configurable datadir to a ZFS filesystem may be better)

I also got a question of minIO integration with Veeam and problems about setup of TLS. If someone has a hint, please post.

I will be abroad until mid march and will do tests myself then. An S3 server ex for Veeam and Duplicati backups seems be a perfect add on to ZFS especially when it works as trouble free as OmniOS does.

As a first howto, you may read the following blog
(Ubuntu + ZFS but may be helpful despite)

DIY cloud backup: Installing and configuring the server - Intermittent Technology
DIY Cloud Backup: Installing and configuring a client - Intermittent Technology

using TLS on Clients:
Windows service + minio with ssl error

 

[gea]

To install minIO ex on OmniOS 151032:
(Server, client and rclone, the rsync for Cloud)
For 151030 or bloody use their extra repository

pkg set-publisher -g https://pkg.omniosce.org/r151032/extra extra.omnios

pkg install minio
pkg install minio-mc
pkg install rclone

#enable minio as a service
svcadm enable minio

If you want to start minio manually with command line options:
- disable service: svcadm disable minio (check service state via svcs)
- binaries are in /opt/ooce
ex minIO itself
/opt/ooce/minio/bin/minio

Start minIO

Start miniO
1.
#Create datadir ex /s3/data and configdir /s3/config

2.
#Start minIO (credidentials must be set via export)
export MINIO_ACCESS_KEY=minio;
export MINIO_SECRET_KEY=minio1234;
/opt/ooce/minio/bin/minio server /s3/data --config-dir /s3/config;

3.
#Start Browser ex Google Crome to view, delete, upload, download data via browser
http://ip:9000

enter the above credidentials (minio and minio1234)

If you want to modify accessdata:
This requires to export old and new values ex:

export MINIO_ACCESS_KEY=mini2; 
export MINIO_SECRET_KEY=minio134; 
export MINIO_ACCESS_KEY_OLD=minio; 
export MINIO_SECRET_KEY_OLD=minio133; 
/opt/ooce/minio/bin/minio server /s3/data --config-dir /s3/config;

In next napp-it I plan to add an S3 sharing option per filesystem as datadir just like NFS or SMB

 

[asche]

Pardon me, but what's the use case at home/single system, e.g. for Veeam vs. using a simple SMB share? My Veeam agents all save to a SMB share hosted on my OmniOS NAS ...?

I can see the benefit if minIO is used to provide distributed storage (a la ceph), but that's a different kettle of fish.

 

[ma7c]

Hi Asche,

the reasons why i would recommend S3 over SMB are shown in the following article: in a case of desaster all accessible shares in a active directory can by encrypted by a sophisticated locker. One real-life example is the story of Norsk Hydro in march last year: How Lockergoga took down Hydro — ransomware used in targeted attacks aimed at big business

The recommendation from the article: use read-only Backups (!) = solved by ZFS-Snaps and a heavily defended backup infrastructure: user credentials of Active Directory users can not be used to access backups by SMB or even better S3! (all your accessible SMB shares can be encrypted).

• Backup everything.
• Have read only backups, too, if you use disk-to-disk.
• Have a very heavily defended backup infrastructure. Only the people who must have access to a backup server should have access. Take Domain Admins out the Local Administrator group; make it truly secure.

Marc

Pardon me, but what's the use case at home/single system, e.g. for Veeam vs. using a simple SMB share? My Veeam agents all save to a SMB share hosted on my OmniOS NAS ...?

I can see the benefit if minIO is used to provide distributed storage (a la ceph), but that's a different kettle of fish.

 

[asche]

Thanks @ma7c! However, just to note that for Veeam you can use a separate SMB share with separate credentials, so the cryptolocker would need to either sniff the password or pick it up from the Veeam executable/files.

 

[ma7c]

Yes, you are right, getting the credentials can happen, directly from a VEEAM Windows Server:

I a case of an attack, those lockers can get access to the backup server too (they are, in most cases, member of the AD-Infrastructure). Then the Credential Manager is your last layer of defense.
The VEEAM Credential Manager, where all those logins (of your NAS SMB Repo) are saved, is not that safe place in that case...

"Yes, it is definitely a possibility. This would require a hacker to wait for the next zero-day privilege escalation vulnerability in Windows, which will enable them to get LOCAL SYSTEM privileges. With that, extracting all saved credentials is trivial - whether from Veeam software, any other software, or Windows Credentials Manager itself."


Source: Gostev, VEEAM PM SVP
Account/Password is safe?​

Thanks @ma7c! However, just to note that for Veeam you can use a separate SMB share with separate credentials, so the cryptolocker would need to either sniff the password or pick it up from the Veeam executable/files.

 

[gea]

NFS, the network file system is simple and fast but totally unsecure, only usable in a secure network. SMB is best for a LAN as it offers authorisation, authentication, file locking and permissions over files and folders. But you should NEVER offer SMB to the Internet.

S3 is the service for the Internet and Cloud. It cannot replace SMB but is ideal as a flat and simple backup destination ex for Veeam, Duplicati or other applications that can backup to Cloud services. Combined with a ZFS pool you can use it in single node mode as an additional sharing option where you can access data directly with snaps, deduplication, encryption and caching or you can build a Cluster where data is distributed.

If you want to use S3 as inhouse backup destination, minIO and ZFS is the lightweight solution that can scale when needed..

 

[gea]

Updated start thread with informations about setup, start and options of minIO

 

[ma7c]

VEEAM had a major update last week and changed the S3 handling. (In version 10.x of VEEAM Backup & Replication).

A Job setting enables S3 compatible Backups immediately as Capacity Tier repository. (So backups are created on the Performance Tier Repo and then copied to the Capacity Tier - our S3 compatible ZFS-MinIO Servers).

 

[gea]

I have added basic support for minIO S3 sharing for OmniOS
as a filesystem property in napp-it 20.dev and 19.12 homeuse

Howto:
- Update napp-it (About > Update)
- Use menu Services > minIO S3 Services to install minIO
- Use menu ZFS filesystems, klick on unset under S3cloud and activate ex on port :9000

You can basically share the same filesystem via SMB and use ZFS versioning or S3 share a file via 1-7 day link that you can create in the minIO webbrowser. ZFS encryption completes this to a perfect solution.

Via SMB, you will find a folder S3_data and S3_config with S3 data .

Care about permissions as you cannot keep them in sync with S3
but it is possible to access the same data via S3 and SMB.

Open a browser (or any S3 client) with address ip:9000

 

[sth]

hi Gea, this is working really well, thank you.
EDIT: Deleted question re interfaces.

 

[gea]

Just to answer if others wants to restrict access as well

The idea behind napp-it S3 sharing is that this should not be a single OS service but work like SMB sharing on a per filesystem base. This requires an instance of minio per filesystem, each listening on a unique ip and/or port.

When you enable S3 sharing for a filesystem, you must enter a name/passwort and the unique port or ip + port combination. This means that you can enter something like

:9000 and :9001 for two filesystems
or 172.17.1.1:9000 and 172.17.1.2:9000 if you want top use the same port on two ip addresses

see

 

[sth]

What sort of performance do you see from your SMB, NFS and Minio shares Gea. I was pleasantly surprised my Minio share hit 150MB/s over wifi (802.11ax/2.5gbps backhauls) which is quite a bit higher than what I've seen with SMB. I wondered if this was consistent with your experience or should I look to debug my slower SMB connection?

 

[gea]

I have not done such tests and I doubt it makes sense. SMB is a filesharing protocol where you can work directly on a share with a full featured filesystem, filelocking, user dependent permissions etc. S3 is object storage without such features only optomized for availability in a cluster environment, performance and scalability to Zetabytes.

If so, you can only compare to other cloud services that offer a simple upload/download/sync and share ex via Apache webserver based tools or a server like Titan that also offers web access but that respects Windows AD permissions. In this case, minIO and S3 seems to be much faster.