Amazon S3 compatible ZFS cloud with minIO

Discussion in 'Solaris, Nexenta, OpenIndiana, and napp-it' started by gea, Feb 17, 2020.

  1. gea

    gea Well-Known Member

    Joined:
    Dec 31, 2010
    Messages:
    2,366
    Likes Received:
    793
    Due a customer/user request, OmniOS added minIO client and server and rclone
    to the extra repo Package Catalog

    MinIO | Enterprise Grade, High Performance Object Storage

    MinIO | MinIO Client Quickstart Guide
    MinIO | The MinIO Quickstart Guide

    MinIO Enjoying Role in Emerging Cloud Architecture

    As backup/sharing client you can use

    - your browser,
    via webclient and http://ip::9000, part of minIO
    - rclone see rclone - rsync for cloud storage
    - Duplicati, a Windows, OSX, Linux backup application ,
    see Duplicati
    - Veeam
    see https://helpcenter.veeam.com/docs/backup/vsphere/adding_s3c_object_storage.html?ver=100


    Install minIO (current OmniOS)
    Code:
    pkg set-publisher -g https://pkg.omniosce.org/r151032/extra extra.omnios
    
    pkg install minio
    pkg install minio-mc
    pkg install rclone
    
    #optionally enable minio as a service
    # (I prefer a manual start with options, you can start multiple instances with a different port)
    svcadm enable minio
    
    # service manifest, see /lib/svc/manifest/application/application-minio.xml
    

    Start minIO manually
    Code:
    1.
    #Create datadir ex /s3/data and configdir /s3/config
    
    2.
    #Start minIO (credidentials must be set via export)
    export MINIO_ACCESS_KEY=minio;
    export MINIO_SECRET_KEY=minio1234;
    /opt/ooce/minio/bin/minio server /s3/data --config-dir /s3/config;
    
    3.
    #Start Browser ex Google Crome to view, delete, upload, download data via browser
    http://ip:9000
    
    enter the above credidentials (ex minio and minio1234)

    If you want to modify accessdata:
    This requires to export old and new values ex:
    Code:
    export MINIO_ACCESS_KEY=mini2;
    export MINIO_SECRET_KEY=minio134;
    export MINIO_ACCESS_KEY_OLD=minio;
    export MINIO_SECRET_KEY_OLD=minio133;
    /opt/ooce/minio/bin/minio server /s3/data --config-dir /s3/config;

    Start options of miniIO
    Code:
    USAGE:
      minio [FLAGS] COMMAND [ARGS...]
    
    COMMANDS:
      server   start object storage server
      gateway  start object storage gateway
    
    FLAGS:
      --config-dir value, -C value  [DEPRECATED] path to legacy configuration directory (default: "/root/.minio")
      --address value               bind to a specific ADDRESS:PORT, ADDRESS can be an IP or hostname (default: ":9000")
      --certs-dir value, -S value   path to certs directory (default: "/root/.minio/certs")
      --quiet                       disable startup information
      --anonymous                   hide sensitive information from logging
      --json                        output server logs and startup information in json format
      --compat                      enable strict S3 compatibility by turning off certain performance optimizations
      --help, -h                    show help
      --version, -v                 print the version
    

    Windows and TLS
    see Windows service + minio with ssl error


    Optional: start minIO in distributed/ clustered mode
    Code:
    output from:  ./minio server
    
    EXAMPLES:
      1. Start minio server on "/home/shared" directory.
         $ minio server /home/shared
    
      2. Start distributed minio server on an 32 node setup with 32 drives each, run following command on all the nodes
         $ export MINIO_ACCESS_KEY=minio
         $ export MINIO_SECRET_KEY=miniostorage
         $ minio server http://node{1...32}.example.com/mnt/export/{1...32}
    
      3. Start distributed minio server in an expanded setup, run the following command on all the nodes
         $ export MINIO_ACCESS_KEY=minio
         $ export MINIO_SECRET_KEY=miniostorage
         $ minio server http://node{1...16}.example.com/mnt/export/{1...32} \
                http://node{17...64}.example.com/mnt/export/{1...64}
    
    
     
    #1
    Last edited: Feb 21, 2020
    fasting, sth, Patrick and 3 others like this.
  2. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    4,020
    Likes Received:
    692
    Ok, just for my understanding - this allows me to run a S3 compatible storage at home on my Napp-It box?
    So whenever anything touts S3 integration I can run that against my own box?

    That would be effing brilliant for all those stupid (to me) cloud (only) integrated apps nowadays.
     
    #2
  3. gea

    gea Well-Known Member

    Joined:
    Dec 31, 2010
    Messages:
    2,366
    Likes Received:
    793
    Yes, this is the point, an inhouse replacement of S3 as an addition to a ZFS filer.
    OmniOS added minIO this week due a request for an S3 compatible backup destination for a Veeam environment. This makes a huge difference over compiling yourself.

    I have not tried minIO myself, this will be the next step after encryption is integrated.
    As far as I see, all what is needed is a basic Howto, a setup script (wget..) that installs miniIO, downloads rclone and does a basic setup. Within napp-it propably a menu for service management, configuration and optionally Cluster management if more than one miniIO server is intended. Just to make it as easy as the ZFS filer.

    In a second step I may include it then in a basic napp-it setup as a default feature.
     
    #3
    Last edited: Feb 19, 2020
  4. StevenDTX

    StevenDTX Active Member

    Joined:
    Aug 17, 2016
    Messages:
    332
    Likes Received:
    125
    I have used Minio for years. I installed Duplicati on all of my family's computers and they all back up to me.
     
    #4
  5. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    4,020
    Likes Received:
    692
    That sounds great :)

    Anything that can do Dropbox too?
     
    #5
  6. gregsachs

    gregsachs Active Member

    Joined:
    Aug 14, 2018
    Messages:
    265
    Likes Received:
    68
    Ditto, I run my offsite backup to a Pi with Minio and USB disk.
     
    #6
  7. gea

    gea Well-Known Member

    Joined:
    Dec 31, 2010
    Messages:
    2,366
    Likes Received:
    793
    I have got an info from OmniOS as they now added `rclone` as
    well to the -extra repositories for releases r30/r32/bloody.

    A remark to the minio server package:
    it ships with a default instance using /var/opt/ooce/minio as its data
    root (in accordance to their -extra packages layout).
    however new SMF instances can be created and the SMF `datadir` be set to
    a different dataset/path. (A configurable datadir to a ZFS filesystem may be better)

    I also got a question of minIO integration with Veeam and problems about setup of TLS. If someone has a hint, please post.

    I will be abroad until mid march and will do tests myself then. An S3 server ex for Veeam and Duplicati backups seems be a perfect add on to ZFS especially when it works as trouble free as OmniOS does.

    As a first howto, you may read the following blog
    (Ubuntu + ZFS but may be helpful despite)

    DIY cloud backup: Installing and configuring the server - Intermittent Technology
    DIY Cloud Backup: Installing and configuring a client - Intermittent Technology

    using TLS on Clients:
    Windows service + minio with ssl error
     
    #7
    Last edited: Feb 19, 2020
  8. gea

    gea Well-Known Member

    Joined:
    Dec 31, 2010
    Messages:
    2,366
    Likes Received:
    793
    To install minIO ex on OmniOS 151032:
    (Server, client and rclone, the rsync for Cloud)
    For 151030 or bloody use their extra repository

    Code:
    pkg set-publisher -g https://pkg.omniosce.org/r151032/extra extra.omnios
    
    pkg install minio
    pkg install minio-mc
    pkg install rclone
    
    #enable minio as a service
    svcadm enable minio
    
    If you want to start minio manually with command line options:
    - disable service: svcadm disable minio (check service state via svcs)
    - binaries are in /opt/ooce
    ex minIO itself
    /opt/ooce/minio/bin/minio

    Start minIO
    Code:
    Start miniO
    1.
    #Create datadir ex /s3/data and configdir /s3/config
    
    2.
    #Start minIO (credidentials must be set via export)
    export MINIO_ACCESS_KEY=minio;
    export MINIO_SECRET_KEY=minio1234;
    /opt/ooce/minio/bin/minio server /s3/data --config-dir /s3/config;
    
    3.
    #Start Browser ex Google Crome to view, delete, upload, download data via browser
    http://ip:9000
    
    enter the above credidentials (minio and minio1234)
    If you want to modify accessdata:
    This requires to export old and new values ex:

    Code:
    export MINIO_ACCESS_KEY=mini2; 
    export MINIO_SECRET_KEY=minio134; 
    export MINIO_ACCESS_KEY_OLD=minio; 
    export MINIO_SECRET_KEY_OLD=minio133; 
    /opt/ooce/minio/bin/minio server /s3/data --config-dir /s3/config;
    In next napp-it I plan to add an S3 sharing option per filesystem as datadir just like NFS or SMB
     
    #8
    Last edited: Feb 21, 2020
    Patrick and Rand__ like this.
  9. asche

    asche New Member

    Joined:
    Oct 6, 2017
    Messages:
    17
    Likes Received:
    3
    Pardon me, but what's the use case at home/single system, e.g. for Veeam vs. using a simple SMB share? My Veeam agents all save to a SMB share hosted on my OmniOS NAS ...?

    I can see the benefit if minIO is used to provide distributed storage (a la ceph), but that's a different kettle of fish.
     
    #9
  10. ma7c

    ma7c New Member

    Joined:
    Feb 19, 2020
    Messages:
    3
    Likes Received:
    3
    Hi Asche,

    the reasons why i would recommend S3 over SMB are shown in the following article: in a case of desaster all accessible shares in a active directory can by encrypted by a sophisticated locker. One real-life example is the story of Norsk Hydro in march last year: How Lockergoga took down Hydro — ransomware used in targeted attacks aimed at big business

    The recommendation from the article: use read-only Backups (!) = solved by ZFS-Snaps and a heavily defended backup infrastructure: user credentials of Active Directory users can not be used to access backups by SMB or even better S3! (all your accessible SMB shares can be encrypted).

    • Backup everything.
    • Have read only backups, too, if you use disk-to-disk.
    • Have a very heavily defended backup infrastructure. Only the people who must have access to a backup server should have access. Take Domain Admins out the Local Administrator group; make it truly secure.

    Marc

     
    #10
    asche likes this.
  11. asche

    asche New Member

    Joined:
    Oct 6, 2017
    Messages:
    17
    Likes Received:
    3
    Thanks @ma7c! However, just to note that for Veeam you can use a separate SMB share with separate credentials, so the cryptolocker would need to either sniff the password or pick it up from the Veeam executable/files.
     
    #11
  12. ma7c

    ma7c New Member

    Joined:
    Feb 19, 2020
    Messages:
    3
    Likes Received:
    3
    Yes, you are right, getting the credentials can happen, directly from a VEEAM Windows Server:

    I a case of an attack, those lockers can get access to the backup server too (they are, in most cases, member of the AD-Infrastructure). Then the Credential Manager is your last layer of defense.
    The VEEAM Credential Manager, where all those logins (of your NAS SMB Repo) are saved, is not that safe place in that case...

    "Yes, it is definitely a possibility. This would require a hacker to wait for the next zero-day privilege escalation vulnerability in Windows, which will enable them to get LOCAL SYSTEM privileges. With that, extracting all saved credentials is trivial - whether from Veeam software, any other software, or Windows Credentials Manager itself."


    Source: Gostev, VEEAM PM SVP
    Account/Password is safe?

     
    #12
    Last edited: Feb 19, 2020
  13. gea

    gea Well-Known Member

    Joined:
    Dec 31, 2010
    Messages:
    2,366
    Likes Received:
    793
    NFS, the network file system is simple and fast but totally unsecure, only usable in a secure network. SMB is best for a LAN as it offers authorisation, authentication, file locking and permissions over files and folders. But you should NEVER offer SMB to the Internet.

    S3 is the service for the Internet and Cloud. It cannot replace SMB but is ideal as a flat and simple backup destination ex for Veeam, Duplicati or other applications that can backup to Cloud services. Combined with a ZFS pool you can use it in single node mode as an additional sharing option where you can access data directly with snaps, deduplication, encryption and caching or you can build a Cluster where data is distributed.

    If you want to use S3 as inhouse backup destination, minIO and ZFS is the lightweight solution that can scale when needed..
     
    #13
    Last edited: Feb 19, 2020
  14. gea

    gea Well-Known Member

    Joined:
    Dec 31, 2010
    Messages:
    2,366
    Likes Received:
    793
    Updated start thread with informations about setup, start and options of minIO
     
    #14
    sth and Patrick like this.
  15. ma7c

    ma7c New Member

    Joined:
    Feb 19, 2020
    Messages:
    3
    Likes Received:
    3
    VEEAM had a major update last week and changed the S3 handling. (In version 10.x of VEEAM Backup & Replication).

    A Job setting enables S3 compatible Backups immediately as Capacity Tier repository. (So backups are created on the Performance Tier Repo and then copied to the Capacity Tier - our S3 compatible ZFS-MinIO Servers).

    [​IMG]
     
    #15
    gea and Evan like this.
  16. gea

    gea Well-Known Member

    Joined:
    Dec 31, 2010
    Messages:
    2,366
    Likes Received:
    793
    I have added basic support for minIO S3 sharing for OmniOS
    as a filesystem property in napp-it 20.dev and 19.12 homeuse

    Howto:
    - Update napp-it (About > Update)
    - Use menu Services > minIO S3 Services to install minIO
    - Use menu ZFS filesystems, klick on unset under S3cloud and activate ex on port :9000

    You can basically share the same filesystem via SMB and use ZFS versioning or S3 share a file via 1-7 day link that you can create in the minIO webbrowser. ZFS encryption completes this to a perfect solution.

    Via SMB, you will find a folder S3_data and S3_config with S3 data .

    Care about permissions as you cannot keep them in sync with S3
    but it is possible to access the same data via S3 and SMB.

    Open a browser (or any S3 client) with address ip:9000

    minio.png
     
    #16
    Last edited: Mar 18, 2020
    Patrick, sth, Bronko and 1 other person like this.
  17. sth

    sth Active Member

    Joined:
    Oct 29, 2015
    Messages:
    270
    Likes Received:
    41
    hi Gea, this is working really well, thank you.
    EDIT: Deleted question re interfaces.
     
    #17
    Last edited: Mar 19, 2020
  18. gea

    gea Well-Known Member

    Joined:
    Dec 31, 2010
    Messages:
    2,366
    Likes Received:
    793
    Just to answer if others wants to restrict access as well

    The idea behind napp-it S3 sharing is that this should not be a single OS service but work like SMB sharing on a per filesystem base. This requires an instance of minio per filesystem, each listening on a unique ip and/or port.

    When you enable S3 sharing for a filesystem, you must enter a name/passwort and the unique port or ip + port combination. This means that you can enter something like

    :9000 and :9001 for two filesystems
    or 172.17.1.1:9000 and 172.17.1.2:9000 if you want top use the same port on two ip addresses

    see
    minio.png
     
    #18
    Last edited: Mar 19, 2020
    Patrick likes this.
  19. sth

    sth Active Member

    Joined:
    Oct 29, 2015
    Messages:
    270
    Likes Received:
    41
    What sort of performance do you see from your SMB, NFS and Minio shares Gea. I was pleasantly surprised my Minio share hit 150MB/s over wifi (802.11ax/2.5gbps backhauls) which is quite a bit higher than what I've seen with SMB. I wondered if this was consistent with your experience or should I look to debug my slower SMB connection?
     
    #19
  20. gea

    gea Well-Known Member

    Joined:
    Dec 31, 2010
    Messages:
    2,366
    Likes Received:
    793
    I have not done such tests and I doubt it makes sense. SMB is a filesharing protocol where you can work directly on a share with a full featured filesystem, filelocking, user dependent permissions etc. S3 is object storage without such features only optomized for availability in a cluster environment, performance and scalability to Zetabytes.

    If so, you can only compare to other cloud services that offer a simple upload/download/sync and share ex via Apache webserver based tools or a server like Titan that also offers web access but that respects Windows AD permissions. In this case, minIO and S3 seems to be much faster.
     
    #20
    Last edited: Mar 19, 2020

Share This Page