Configuring Alluxio with Ceph
This guide describes how to configure Alluxio with Ceph as the under storage system. Alluxio supports two different clients APIs to connect to Ceph Object Storage using Rados Gateway:
Initial Setup
First, the Alluxio binaries must be on your machine. You can either compile Alluxio, or download the binaries locally.
Configuring Alluxio
You need to configure Alluxio to use under storage systems by modifying
conf/alluxio-site.properties
. If it does not exist, create the configuration file from the
template.
cp conf/alluxio-site.properties.template conf/alluxio-site.properties
Option 1: S3A Interface (preferred)
Modify conf/alluxio-site.properties
to include:
alluxio.underfs.address=s3a://<bucket>/<folder>
aws.accessKeyId=<access-key>
aws.secretKey=<secret-key>
alluxio.underfs.s3.endpoint=http://<rgw-hostname>:<rgw-port>
alluxio.underfs.s3.disable.dns.buckets=true
alluxio.underfs.s3a.inherit_acl=<inherit-acl>
If using a Ceph release such as hammer (or older) specify alluxio.underfs.s3a.signer.algorithm=S3SignerType
to use v2 S3 signatures. To use GET Bucket (List Objects) Version 1 specify
alluxio.underfs.s3a.list.objects.v1=true
.
Option 2: Swift Interface
Modify conf/alluxio-site.properties
to include:
alluxio.underfs.address=swift://<swift-container>
Where <swift-container>
is an existing Swift container.
The following configuration should be provided in the conf/alluxio-site.properties
fs.swift.user=<swift-user>
fs.swift.tenant=<swift-tenant>
fs.swift.password=<swift-user-password>
fs.swift.auth.url=<swift-auth-url>
fs.swift.use.public.url=<swift-use-public>
fs.swift.auth.method=<swift-auth-model>
Possible values of <swift-use-public>
are true
, false
. Specify <swift-auth-model>
as
swiftauth
if using native Ceph RGW authentication and <swift-auth-url>
as http://<rgw-hostname>:8090/auth/1.0
.
Alternatively, these configuration settings can be set in the conf/alluxio-env.sh
file. More
details about setting configuration parameters can be found in
Configuration Settings.
Running Alluxio Locally with Ceph
After everything is configured, you can start up Alluxio locally to see that everything works.
./bin/alluxio format
./bin/alluxio-start.sh local
This should start an Alluxio master and an Alluxio worker. You can see the master UI at http://localhost:19999.
Next, you can run a simple example program:
./bin/alluxio runTests
After this succeeds, you can visit your bucket/container to verify the files and directories created by Alluxio exist.
If using the S3A connector, you should see files named like:
S3_BUCKET/S3_DIRECTORY/alluxio/data/default_tests_files/Basic_CACHE_THROUGH
If using the Swift connector, you should see files named like:
swift://<SWIFT CONTAINER>/alluxio/data/default_tests_files/Basic_CACHE_THROUGH
To stop Alluxio, you can run:
./bin/alluxio-stop.sh local
Access Control
If Alluxio security is enabled, Alluxio enforces the access control inherited from underlying Ceph Object Storage. Depending on the interace used, refer to S3A Access Control or Swift Access Control for more information.