Configuring Alluxio with MapR-FS
- Compiling Alluxio with MapR Version
- Configuring Alluxio for MapR-FS
- Configuring Alluxio to use MapR-FS as Under File System
- Running Alluxio Locally with MapR-FS
This guide describes how to configure Alluxio with MapR-FS as the under storage system.
Compiling Alluxio with MapR Version
Alluxio must be compiled with the correct MapR distribution
to integrate with MapR-FS. Here are some values of hadoop.version
for different MapR
distributions:
MapR Version | hadoop.version |
---|---|
5.2 | 2.7.0-mapr-1607 |
5.1 | 2.7.0-mapr-1602 |
5.0 | 2.7.0-mapr-1506 |
4.1 | 2.5.1-mapr-1503 |
4.0.2 | 2.5.1-mapr-1501 |
4.0.1 | 2.4.1-mapr-1408 |
Configuring Alluxio for MapR-FS
Once you have compiled Alluxio with the appropriate hadoop.version
for your MapR distribution, you
may have to configure Alluxio to recognize the MapR-FS scheme and URIs. Alluxio uses the HDFS client
to access MapR-FS, and by default is already configured to do so. However, if the configuration has
been changed, you can enable the HDFS client to access MapR-FS URIs by adding the URI prefix
maprfs:///
to the configuration variable alluxio.underfs.hdfs.prefixes
like below:
alluxio.underfs.hdfs.prefixes=hdfs://,maprfs:///
This configuration parameter should be set for all the Alluxio servers (masters, workers). Please
read how to configure Alluxio. For Alluxio processes, this parameter
can be set in the property file alluxio-site.properties
. For more information, please read about
configuration of Alluxio with property files.
This parameter should also be set any client that accesses Alluxio. This means the parameter should
be set for any application (MapReduce, Spark, Flink, etc.) that accesses Alluxio. This can typically
be done by adding -Dalluxio.underfs.hdfs.prefixes=hdfs://,maprfs:///
to the command. For more
information, please read about configurating applications for Alluxio.
Configuring Alluxio to use MapR-FS as Under File System
There are various ways to configure Alluxio to use MapR-FS as the Under File System. If you want to
mount MapR-FS to the root of Alluxio, add the following to conf/alluxio-site.properties
:
alluxio.underfs.address=maprfs:///<path in MapR-FS>/
You can also mount a directory in MapR-FS to a directory in the Alluxio namespace.
${ALLUXIO_HOME}/bin/alluxio fs mount /<path in Alluxio>/ maprfs:///<path in MapR-FS>/
Running Alluxio Locally with MapR-FS
After everything is configured, you can start up Alluxio locally to see that everything works.
./bin/alluxio format
./bin/alluxio-start.sh local
This should start one Alluxio master and one Alluxio worker locally. You can see the master UI at http://localhost:19999.
Next, you can run a simple example program:
./bin/alluxio runTests
After this succeeds, you can visit MapR-FS web UI to verify the files and directories created by
Alluxio exist. For this test, you should see files named like:
/default_tests_files/Basic_CACHE_THROUGH
You can stop Alluxio any time by running:
./bin/alluxio-stop.sh local