Configuring Alluxio with OSS

Slack Docker Pulls

This guide describes how to configure Alluxio with Aliyun OSS as the under storage system. Object Storage Service (OSS) is a massive, secure and highly reliable cloud storage service provided by Aliyun.

Mounting OSS

Alluxio unifies access to different storage systems through the unified namespace feature. An OSS location can be either mounted at the root of the Alluxio namespace or at a nested directory.

Root Mount

When installing, the under storage address and credentials should be specified in conf/alluxio-site.properties.

alluxio.underfs.address=oss://<OSS_BUCKET>/<OSS_DIRECTORY>
fs.oss.accessKeyId=<OSS_ACCESS_KEY_ID>
fs.oss.accessKeySecret=<OSS_ACCESS_KEY_SECRET>
fs.oss.endpoint=<OSS_ENDPOINT>

Here fs.oss.accessKeyId is the Access Key Id string and fs.oss.accessKeySecret is the Access Key Secret string, which are managed in AccessKeys in Aliyun UI. fs.oss.endpoint is the endpoint of this bucket, which can be found in the Bucket overview with possible values like “oss-us-west-1.aliyuncs.com”, “oss-cn-shanghai.aliyuncs.com” (OSS Internet Endpoint).

Nested Mount

An OSS location can be mounted at a nested directory in the Alluxio namespace to have unified access to multiple under storage systems. Alluxio’s Command Line Interface can be used for this purpose.

$ ./bin/alluxio fs mount --option fs.oss.accessKeyId=<OSS_ACCESS_KEY_ID> \
  --option fs.oss.accessKeySecret=<OSS_ACCESS_KEY_SECRET> \
  --option fs.oss.endpoint=<OSS_ENDPOINT> \
  <ALLUXIO_PATH> oss://<OSS_BUCKET>/<OSS_DIRECTORY>

Running Alluxio Locally with OSS

Tests can be run using the Alluxio Command Line Interface.

$ ./bin/alluxio runTests

If testing a nested mount point, run:

$ ./bin/alluxio runTests --directory /mnt/

After this succeeds, in OSS directory OSS_BUCKET/OSS_DIRECTORY verify the files and directories created by Alluxio exist. For this test, you should see files named like:

OSS_BUCKET/OSS_DIRECTORY/default_tests_files/Basic_CACHE_PROMOTE_CACHE_THROUGH