Configuring Alluxio with Minio

• Initial Setup
• Setup Minio
• Configuring Alluxio

This guide describes how to configure Alluxio with Minio as the under storage system. Alluxio natively provides the s3a:// scheme (recommended for better performance). You can use this scheme to connect Alluxio with Minio server.

Initial Setup

First, the Alluxio binaries must be on your machine. You can either compile Alluxio, or download the binaries locally.

Setup Minio

You need to configure Alluxio to use under storage systems by modifying conf/alluxio-site.properties. If it does not exist, create the configuration file from the template.

cp conf/alluxio-site.properties.template conf/alluxio-site.properties

Minio is an object storage server built for cloud applications and DevOps. Minio provides an open source alternative to AWS S3.

Launch a Minio server instance using the steps mentioned here. Then create a bucket (or use an existing bucket). Once the server is launched, keep a note of Minio server endpoint, accessKey and secretKey.

You should also note the directory you want to use in that bucket, either by creating a new directory in the bucket, or using an existing one. For the purposes of this guide, the Minio bucket name is called MINIO_BUCKET, and the directory in that bucket is called MINIO_DIRECTORY.

Configuring Alluxio

You need to configure Alluxio to use Minio as its under storage system by modifying conf/alluxio-site.properties. The first modification is to specify an existing Minio bucket and directory as the under storage system.

All the fields to be modified in conf/alluxio-site.properties file are listed here:

alluxio.underfs.address=s3a://<MINIO_BUCKET>/<MINIO_DIRECTORY>
alluxio.underfs.s3.endpoint=http://<MINIO_ENDPOINT>/
alluxio.underfs.s3.disable.dns.buckets=true
alluxio.underfs.s3a.inherit_acl=false
alluxio.underfs.s3.proxy.https.only=<USE_HTTPS>
alluxio.underfs.s3.endpoint.http.port=<MINIO_HTTP_PORT>
alluxio.underfs.s3.endpoint.https.port=<MINIO_HTTPS_PORT>
aws.accessKeyId=<MINIO_ACCESS_KEY_ID>
aws.secretKey=<MINIO_SECRET_KEY_ID>

For these parameters, replace <MINIO_ENDPOINT>and <MINIO_PORT> with the URL and port of your Minio service.

Replace <USE_HTTPS> with true or false. If true (using HTTPS), also replace <MINIO_HTTPS_PORT>, with the HTTPS port for the provider and remove the alluxio.underfs.s3.endpoint.http.port parameter. If you replace <USE_HTTPS> with false (using HTTP) also replace <MINIO_HTTP_PORT> with the HTTP port for the provider, and remove the alluxio.underfs.s3.endpoint.https.port parameter. If the HTTP or HTTPS port values are left unset, <HTTP_PORT> defaults to port 80, and <HTTPS_PORT> defaults to port 443.