Minio

Slack Docker Pulls GitHub edit source

This guide describes how to configure Alluxio with Minio as the under storage system. Alluxio natively provides the s3a:// scheme (recommended for better performance). You can use this scheme to connect Alluxio with Minio server.

Initial Setup

The Alluxio binaries must be on your machine. You can either compile Alluxio, or download the binaries locally.

Setup Minio

Configure Alluxio to use under storage systems by modifying conf/alluxio-site.properties. If it does not exist, create the configuration file from the template.

  1. cp conf/alluxio-site.properties.template conf/alluxio-site.properties

Minio is an object storage server built for cloud applications and DevOps. Minio provides an open source alternative to AWS S3.

Launch a Minio server instance using the steps mentioned here. Then create a bucket (or use an existing bucket). Once the server is launched, keep a note of Minio server endpoint, accessKey and secretKey.

You should also note the directory you want to use in that bucket, either by creating a new directory in the bucket, or using an existing one. For the purposes of this guide, the Minio bucket name is called MINIO_BUCKET, and the directory in that bucket is called MINIO_DIRECTORY.

Configuring Alluxio

You need to configure Alluxio to use Minio as its under storage system by modifying conf/alluxio-site.properties. The first modification is to specify an existing Minio bucket and directory as the under storage system.

All the fields to be modified in conf/alluxio-site.properties file are listed here:

  1. alluxio.underfs.address=s3a://<MINIO_BUCKET>/<MINIO_DIRECTORY>
  2. alluxio.underfs.s3.endpoint=http://<MINIO_ENDPOINT>/
  3. alluxio.underfs.s3.disable.dns.buckets=true
  4. alluxio.underfs.s3a.inherit_acl=false
  5. aws.accessKeyId=<MINIO_ACCESS_KEY_ID>
  6. aws.secretKey=<MINIO_SECRET_KEY_ID>

For these parameters, replace <MINIO_ENDPOINT> with the hostname and port of your Minio service, e.g., http://localhost:9000. If the port value is left unset, it defaults to port 80 for http and 443 for https.