Configuring Alluxio with Azure Blob Store

Slack Docker Pulls

This guide describes how to configure Alluxio with Azure Blob Store as the under storage system.

Initial Setup

In preparation for using Azure Blob Store with Alluxio, create a new container in your Azure storage account or use an existing container. You should also note that the directory you want to use in that container, either by creating a new directory in the container, or using an existing one. For the purposes of this guide, the Azure storage account name is called <AZURE_ACCOUNT>, the container in that storage account is called <AZURE_CONTAINER> and the directory in that container is called <AZURE_DIRECTORY>. For more information about Azure storage account, Please see here.

Mounting Azure Blob Store

An Azure Blob Store location can be mounted either at the root of the Alluxio namespace or at a nested directory. Alluxio uses the Hadoop support for Azure blob storage (wasb).

Root Mount

When installing Alluxio, the following properties should be specified in conf/alluxio-site.properties.

alluxio.underfs.address=wasb://<AZURE_CONTAINER>@<AZURE_ACCOUNT>.blob.core.windows.net/<AZURE_DIRECTORY>/
alluxio.master.mount.table.root.option.fs.azure.account.key.<AZURE_ACCOUNT>.blob.core.windows.net=<YOUR ACCESS KEY>

Nested Mount

An Azure blob store location can be mounted at a nested directory in the Alluxio namespace to have unified access to multiple under storage systems. Alluxio’s Command Line Interface can be used for this purpose.

$ ./bin/alluxio fs mount --option fs.azure.account.key.<AZURE_ACCOUNT>.blob.core.windows.net=<AZURE_ACCESS_KEY> \
  /mnt/azure wasb://<AZURE_CONTAINER>@<AZURE_ACCOUNT>.blob.core.windows.net/<AZURE_DIRECTORY>/

Running Alluxio Locally with Azure Blob Store

After configuring Alluxio with Azure Blob Store, follow the rest of the Alluxio installation steps to complete the cluster deployment. To run basic I/O tests, click into the deployed cluster and use the drop down menu in the top right. All the tests should pass if the configurations and setup are correct.

Otherwise, tests can be run using the Alluxio Command Line Interface.

$ ./bin/alluxio runTests

If testing a nested mount point, run:

$ ./bin/alluxio runTests --directory /mnt/

After this succeeds, you can visit your container <AZURE_CONTAINER> to verify the files and directories created by Alluxio exist. For this test, you should see files named like:

<AZURE_CONTAINER>/<AZURE_DIRECTORY>/default_tests_files/BASIC_CACHE_PROMOTE_CACHE_THROUGH