Configuring Alluxio with MapR-FS

• Compiling Alluxio with MapR Version
• Configuring Alluxio for MapR-FS
• Configuring Alluxio to use MapR-FS as Under File System
• Running Alluxio Locally with MapR-FS

This guide describes how to configure Alluxio with MapR-FS as the under storage system.

Compiling Alluxio with MapR Version

Alluxio must be compiled with the correct MapR distribution to integrate with MapR-FS. Here are some values of hadoop.version for different MapR distributions:

Configuring Alluxio for MapR-FS

Once you have compiled Alluxio with the appropriate hadoop.version for your MapR distribution, you may have to configure Alluxio to recognize the MapR-FS scheme and URIs. Alluxio uses the HDFS client to access MapR-FS, and by default is already configured to do so. However, if the configuration has been changed, you can enable the HDFS client to access MapR-FS URIs by adding the URI prefix maprfs:/// to the configuration variable alluxio.underfs.hdfs.prefixes like below:

alluxio.underfs.hdfs.prefixes=hdfs://,maprfs:///

This configuration parameter should be set for all the Alluxio servers (masters, workers). Please read how to configure Alluxio. For Alluxio processes, this parameter can be set in the property file alluxio-site.properties. For more information, please read about configuration of Alluxio with property files.

This parameter should also be set any client that accesses Alluxio. This means the parameter should be set for any application (MapReduce, Spark, Flink, etc.) that accesses Alluxio. This can typically be done by adding -Dalluxio.underfs.hdfs.prefixes=hdfs://,maprfs:/// to the command. For more information, please read about configurating applications for Alluxio.

Configuring Alluxio to use MapR-FS as Under File System

There are various ways to configure Alluxio to use MapR-FS as the Under File System. If you want to mount MapR-FS to the root of Alluxio, add the following to conf/alluxio-site.properties:

alluxio.underfs.address=maprfs:///<path in MapR-FS>/

You can also mount a directory in MapR-FS to a directory in the Alluxio namespace.

${ALLUXIO_HOME}/bin/alluxio fs mount /<path in Alluxio>/ maprfs:///<path in MapR-FS>/

Running Alluxio Locally with MapR-FS

After everything is configured, you can start up Alluxio locally to see that everything works.

./bin/alluxio format
./bin/alluxio-start.sh local

This should start one Alluxio master and one Alluxio worker locally. You can see the master UI at http://localhost:19999.

Next, you can run a simple example program:

./bin/alluxio runTests

After this succeeds, you can visit MapR-FS web UI to verify the files and directories created by Alluxio exist. For this test, you should see files named like: /default_tests_files/Basic_CACHE_THROUGH

You can stop Alluxio any time by running:

./bin/alluxio-stop.sh local