Multiple HDFS as Under Store

Slack Docker Pulls

This page describes how to mount multiple HDFS clusters—possibly with different version numbers and/or from different distributions—as under storage to a single Alluxio namespace. With this feature, Alluxio can mask the incompatibility across different HDFS versions (e.g., HDFS 1.x has a different wire protocol from HDFS 2.x) and enable applications to access data in different HDFS clusters seamlessly.

Mount HDFS with Specific Versions

There are multiple ways for a user to mount an HDFS cluster with a specified version as an under storage into Alluxio namespace.

Using Mount Command-line

When using the mount Alluxio shell command, one can pass through the mount option alluxio.underfs.hdfs.version to specify which version of HDFS to mount. If no such a version is specificed, by default Alluxio treats it as Apache HDFS 2.2.

For example, the following commands mount two HDFS deployments—one is HDFS 1.2 and the other is 2.7—into Alluxio namespace under directory /mnt/hdfs12 and /mnt/hdfs27..

$ ./bin/alluxio fs mount --option alluxio.underfs.hdfs.version=1.2 /mnt/hdfs10 hdfs://namenode1:9000/
$ ./bin/alluxio fs mount --option alluxio.underfs.hdfs.version=2.7 /mnt/hdfs27 hdfs://namenode2:9000/

Using Site Properties

When mounting the under storage of Alluxio root directory with a specific HDFS version, one can add the following line to the site properties file (conf/alluxio-site.properties):

alluxio.master.mount.table.root.ufs=hdfs://namenode1:9000
alluxio.master.mount.table.root.option.alluxio.underfs.hdfs.version=1.2

Using Java API

The Alluxio FileSystem client can also set the version in the mount API, by specifying the version in MountOptions.

Example

The following piece of example code achieves the same goal as the previous Alluxio shell command.

// Mount both HDFS clusters to Alluxio
FileSystem fileSystem = FileSystem.Factory.get();
fileSystem.mount("/mnt/hdfs12", "hdfs//namenode1:9000/",
    MountOptions.defaults().setProperty("alluxio.underfs.hdfs.version", "1.2"));
fileSystem.mount("/mnt/hdfs27", "hdfs//namenode2:9000/",
    MountOptions.defaults().setProperty("alluxio.underfs.hdfs.version", "2.7"));

Supported HDFS Versions

Alluxio enterprise-1.8.0 supports the following versions of HDFS as a valid argument of mount option alluxio.underfs.hdfs.version:

  • Apache Hadoop: 1.0, 1.2, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7, 2.8;
  • CDH: cdh5.6, cdh5.8, cdh5.11, cdh5.12, cdh5.13, cdh5.14;
  • HDP: hdp2.4, hdp2.5, hdp2.6;
  • MapRFS: maprfs4.1, maprfs5.0, maprfs5.1, maprfs5.2.