Configuring Alluxio with HDFS

Slack Docker Pulls

This guide describes how to configure Alluxio with HDFS as the under storage system.

Initial Setup

In preparation for using HDFS with Alluxio, have a HDFS cluster available to use.

HDFS Underfs address

The underfs address for HDFS is in the form hdfs://<NAMENODE_ADDRESS>/<PATH>. The path after the namenode address specifies which directory in HDFS to use as the underfs. For example, if the path is /, the root HDFS directory will be mapped to the root Alluxio directory. If the path is /alluxio/data, the file /alluxio/data/file in HDFS will be mapped to the file /file in Alluxio.

Mounting HDFS

Alluxio unifies access to different storage systems through the unified namespace feature. An HDFS cluster can be either mounted at the root of the Alluxio namespace or at a nested directory.

Root Mount

When installing Alluxio, the under storage address and credentials should be specified in conf/alluxio-site.properties.

alluxio.underfs.address=hdfs://<NAMENODE_ADDRESS>/<PATH>
alluxio.master.mount.table.root.option.alluxio.underfs.hdfs.version=<HDFS_VERSION>

Nested Mount

An HDFS cluster can be mounted at a nested directory in the Alluxio namespace to have unified access to multiple under storage systems. Alluxio’s Command Line Interface can be used for this purpose.

$ ./bin/alluxio fs mount --option alluxio.underfs.hdfs.version=<HDFS_VERSION>\
  /mnt/hdfs hdfs://<NAMENODE_ADDRESS>/<PATH>

If --option alluxio.underfs.hdfs.version=<HDFS_VERSION> is not specified, Alluxio assumes the HDFS version is from property alluxio.underfs.hdfs.version if it is specified in conf/alluxio-site.properties or “2.2” otherwise.

Running Alluxio Locally with HDFS

Tests can be run using the Alluxio Command Line Interface.

$ ./bin/alluxio runTests

After this succeeds, you can visit HDFS web UI (by default at http://localhost:50070) to verify the files and directories created by Alluxio exist. For this test, you should see files named like:

/default_tests_files/BASIC_CACHE_THROUGH

Advanced Configurations

Additional properties can be specified by modifying conf/alluxio-site.properties.