Building Alluxio From Source
This guide describes how to clone the Alluxio repository, compile the source code, and run tests in your environment.
- Required Software
- Checkout Source Code
- Build Options
Checkout Source Code
Checkout the Alluxio master branch from Github:
$ git clone git://github.com/alluxio/alluxio.git $ cd alluxio
By default, cloning the repository will check out the master branch. If you are looking to build a particular version of the code you may check out the version using a git tag.
$ git tag $ git checkout <TAG_NAME>
Build the source code using Maven:
$ mvn clean install -DskipTests
To speed up the compilation, you can run the following instruction to skip different checks:
$ mvn -T 2C clean install -DskipTests -Dmaven.javadoc.skip -Dfindbugs.skip \ -Dcheckstyle.skip -Dlicense.skip -Dskip.protoc
Note: The flag
-Dskip.protocskips generating source files related to gRPC proto. You can skip this step if you have already built them.
If you are seeing
java.lang.OutOfMemoryError: Java heap space, please set the following
variable to increase the memory heap size for maven:
$ export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"
The Maven build system fetches its dependencies, compiles source code, runs unit tests, and packages the system. If this is the first time you are building the project, it can take a while to download all the dependencies. Subsequent builds, however, will be much faster.
Once Alluxio is built, you can validate and start it with:
$ # Alluxio uses ./underFSStorage for under file system storage by default $ mkdir ./underFSStorage $ ./bin/alluxio validateEnv local $ ./bin/alluxio format $ ./bin/alluxio-start.sh local SudoMount
To verify that Alluxio is running, you can visit http://localhost:19999 or
check the log in the
alluxio/logs directory. The
master.log files will
typically be the most useful. It may take a few seconds for the web server to start. You can also
run a simple program to test that data can be read and written to Alluxio’s UFS:
$ ./bin/alluxio runTests
You should be able to see the result
Passed the test!
You can stop the local Alluxio system by using:
$ ./bin/alluxio-stop.sh local
Compute Framework Support
Since Alluxio 1.7, you do not need to run Maven build with different compute profiles.
The Alluxio client jar built and located at
/<PATH_TO_ALLUXIO>/client/alluxio-2.4.0-SNAPSHOT-client.jar will work with different compute frameworks
(e.g., Spark, Flink, Presto and etc) by default.
Hadoop Distribution Support
To build Alluxio against one of the different distributions of hadoop, you can run the following
command by specifying
<HADOOP_PROFILE> and the corresponding
$ mvn install -P<HADOOP_PROFILE> -Dhadoop.version=<HADOOP_VERSION> -DskipTests
<HADOOP_VERSION> can be set for different distributions.
Available Hadoop profiles include
hadoop-3 to cover the major Hadoop
versions 1.x, 2.x and 3.x.
All main builds are from Apache so all Apache releases can be used directly
-Phadoop-1 -Dhadoop.version=1.0.4 -Phadoop-1 -Dhadoop.version=1.2.0 -Phadoop-2 -Dhadoop.version=2.2.0 -Phadoop-2 -Dhadoop.version=2.3.0 -Phadoop-2 -Dhadoop.version=2.4.1 -Phadoop-2 -Dhadoop.version=2.5.2 -Phadoop-2 -Dhadoop.version=2.6.5 -Phadoop-2 -Dhadoop.version=2.7.3 -Phadoop-2 -Dhadoop.version=2.8.0 -Phadoop-2 -Dhadoop.version=2.9.0 -Phadoop-3 -Dhadoop.version=3.0.0
To build against Cloudera’s releases, just use a version like
-Phadoop-2 -Dhadoop.version=2.3.0-cdh5.1.0 -Phadoop-2 -Dhadoop.version=2.0.0-cdh4.7.0
To build against a Hortonworks release, just use a version like
-Phadoop-2 -Dhadoop.version=184.108.40.206.0.5.0-67 -Phadoop-2 -Dhadoop.version=220.127.116.11.1.0.0-92 -Phadoop-2 -Dhadoop.version=18.104.22.168.1.3.0-563