Configuring Alluxio with Minio
This guide describes how to configure Alluxio with Minio as the under storage system. Alluxio natively provides the s3a:// scheme (recommended for better performance). You can use this scheme to connect Alluxio with Minio server.
Initial Setup
First, the Alluxio binaries must be on your machine. You can either compile Alluxio, or download the binaries locally.
Setup Minio
You need to configure Alluxio to use under storage systems by modifying
conf/alluxio-site.properties
. If it does not exist, create the configuration file from the
template.
cp conf/alluxio-site.properties.template conf/alluxio-site.properties
Minio is an object storage server built for cloud applications and DevOps. Minio provides an open source alternative to AWS S3.
Launch a Minio server instance using the steps mentioned here. Then create a bucket (or use an existing bucket). Once the server is launched, keep a note of Minio server endpoint, accessKey and secretKey.
You should also note the directory you want to use in that bucket, either by creating
a new directory in the bucket, or using an existing one. For the purposes of this guide, the Minio bucket name is called
MINIO_BUCKET
, and the directory in that bucket is called MINIO_DIRECTORY
.
Configuring Alluxio
You need to configure Alluxio to use Minio as its under storage system by modifying
conf/alluxio-site.properties
. The first modification is to specify an existing Minio
bucket and directory as the under storage system.
All the fields to be modified in conf/alluxio-site.properties
file are listed here:
alluxio.underfs.address=s3a://<MINIO_BUCKET>/<MINIO_DIRECTORY>
alluxio.underfs.s3.endpoint=http://<MINIO_ENDPOINT>/
alluxio.underfs.s3.disable.dns.buckets=true
alluxio.underfs.s3a.inherit_acl=false
alluxio.underfs.s3.proxy.https.only=<USE_HTTPS>
alluxio.underfs.s3.endpoint.http.port=<MINIO_HTTP_PORT>
alluxio.underfs.s3.endpoint.https.port=<MINIO_HTTPS_PORT>
aws.accessKeyId=<MINIO_ACCESS_KEY_ID>
aws.secretKey=<MINIO_SECRET_KEY_ID>
For these parameters, replace <MINIO_ENDPOINT>
and <MINIO_PORT>
with the URL and port of your Minio service.
Replace <USE_HTTPS>
with true
or false
. If true
(using HTTPS), also replace <MINIO_HTTPS_PORT>
,
with the HTTPS port for the provider and remove the alluxio.underfs.s3.endpoint.http.port
parameter. If you replace <USE_HTTPS>
with false
(using HTTP) also replace <MINIO_HTTP_PORT>
with
the HTTP port for the provider, and remove the alluxio.underfs.s3.endpoint.https.port
parameter.
If the HTTP or HTTPS port values are left unset, <HTTP_PORT>
defaults to port 80, and
<HTTPS_PORT>
defaults to port 443.