Azure Blob Store
- Basic Setup
- Running Alluxio Locally with Azure Blob Store
This guide describes how to configure Alluxio with Azure Blob Store as the under storage system.
The Alluxio binaries must be on your machine.
In preparation for using Azure Blob Store with Alluxio, create a new container in your Azure
storage account or use an existing container. You should also note that the directory you want to
use in that container, either by creating a new directory in the container, or using an existing
one. For the purposes of this guide, the Azure storage account name is called
container in that storage account is called
<AZURE_CONTAINER> and the directory in that container is
<AZURE_DIRECTORY>. For more information about Azure storage account, Please see
To use Azure blob store as the UFS of Alluxio root mount point,
you need to configure Alluxio to use under storage systems by modifying
conf/alluxio-site.properties. If it does not exist, create the configuration file from the
$ cp conf/alluxio-site.properties.template conf/alluxio-site.properties
Specify the underfs address by modifying
conf/alluxio-site.properties to include:
Specify credentials for the Azure account of the root mount point by adding the following
alluxio.master.mount.table.root.option.fs.azure.account.key.<AZURE_ACCOUNT>.blob.core.windows.net=<YOUR ACCESS KEY>
An Azure blob store location can be mounted at a nested directory in the Alluxio namespace to have unified access to multiple under storage systems. Alluxio’s Command Line Interface can be used for this purpose.
$ ./bin/alluxio fs mount \ --option fs.azure.account.key.<AZURE_ACCOUNT>.blob.core.windows.net=<AZURE_ACCESS_KEY> \ /mnt/azure wasb://<AZURE_CONTAINER>@<AZURE_ACCOUNT>.blob.core.windows.net/<AZURE_DIRECTORY>/
After these changes, Alluxio should be configured to work with Azure Blob Store as its under storage system, and you can run Alluxio locally with it.
Running Alluxio Locally with Azure Blob Store
Start up Alluxio locally to see that everything works.
./bin/alluxio format ./bin/alluxio-start.sh local
This should start an Alluxio master and an Alluxio worker. You can see the master UI at http://localhost:19999.
Run a simple example program:
Visit your container
<AZURE_CONTAINER> to verify the files and directories created by Alluxio exist. For this test, you should see files named like:
To stop Alluxio, you can run:
What should I do if I get http not support error?
If you mount the Blob and configure the Blob path start with
wasb://, you may see the error as follows:
alluxio.exception.AlluxioException: com.microsoft.azure.storage.StorageException: The account being accessed does not support http.
You can change the Blob path start with