Contributor Tools

IDE

We recommend using either Eclipse or IntelliJ IDEA to contribute to Alluxio. You can generate an Eclipse configuration file by running:

mvn clean -Pdeveloper install -DskipTests
mvn clean -Pdeveloper -DskipTests eclipse:eclipse -DdownloadJavadocs=true -DdownloadSources=true

Then import the folder into Eclipse.

You may also have to add the classpath variable M2_REPO by running:

mvn -Declipse.workspace="your Eclipse Workspace" eclipse:configure-workspace

If you are using IntelliJ IDEA, you may need to change the Maven profile to ‘developer’ in order to avoid import errors. You can do this by going to

View > Tool Windows > Maven Projects

Maven Targets and Plugins

Before pushing changes or submitting pull requests we recommend running various maven targets on your local machine to make sure your changes do not break existing behavior.

For these maven commands we’ll assume that your command terminal is located in the root directory of your locally cloned copy of the Alluxio repository.

cd ${ALLUXIO_HOME}

Checkstyle

To make sure your code follows our style conventions you may run. Note that this is run any time you run targets such as compile, install, or test.

mvn checkstyle:checkstyle

FindBugs

Before submitting the pull-request, run the latest code against the findbugs Maven plugin to verify no new warnings are introduced.

mvn compile findbugs:findbugs

Compilation

To simply compile the code you can run the following command:

mvn clean compile -DskipTests

This will not execute any unit tests but will execute the checkstyle, findbugs, and other plugins.

To speed up compilation you may use the command:

mvn -T 2C compile -DskipTests -Dmaven.javadoc.skip -Dfindbugs.skip -Dcheckstyle.skip -Dlicense.skip

This command will skip many of our checks that are in place to help keep our code neat. We recommend running all checks before committing.

You may replace the compile target in the above command with any other valid target to skip checks as well. The targets install, verify, and compile will be most useful.

Creating a Local Install

If you want to test your changes with a compiled version of the repository, you may generate the jars with the Maven install target.

mvn install -DskipTests

After the install target executes, you may configure and start a local cluster with the following commands:

If you haven’t configured or set up a local cluster yet, run the following commands

cp conf/alluxio-site.properties.template conf/alluxio-site.properties
echo "alluxio.master.hostname=localhost" >> conf/alluxio-site.properties
./bin/alluxio format

Once you’ve run those configuration steps you can start a local Alluxio instance with

./bin/alluxio-start.sh local SudoMount

Unit Tests

  • Run all unit and integration tests
cd ${ALLUXIO_HOME}
mvn test

This will use the local filesystem as the under storage.

  • Run a single unit test:
mvn -Dtest=AlluxioFSTest#createFileTest -DfailIfNoTests=false test
  • To run unit tests for a specific module, execute the maven test command targeting the desired submodule directory. For example, to run tests for HDFS UFS module you would run
mvn test -pl underfs/hdfs

Run unit tests for HDFS UFS module with a different Hadoop version:

# build and run test on HDFS under storage module for Hadoop 2.7.0
mvn test -pl underfs/hdfs -Phadoop-2 -Dhadoop.version=2.7.0
# build and run test on HDFS under storage module for Hadoop 3.0.0
mvn test -pl underfs/hdfs -Phadoop-3 -Dhadoop.version=3.0.0

The above unit tests will create a simulated HDFS service with the specific version. To run more comprehensive tests on HDFS under storage using a real and running HDFS deployment:

mvn test -pl underfs/hdfs -PufsContractTest -DtestHdfsBaseDir=hdfs://ip:port/alluxio_test
  • To have the logs output to STDOUT, append the following arguments to the mvn command
-Dtest.output.redirect=false -Dalluxio.root.logger=DEBUG,CONSOLE
  • To quickly test the working of some APIs in an interactive manner, you may leverage the Scala shell, as discussed in this blog.

  • The fuse tests are ignored if the libfuse library is missing. To run those tests, please install the correct libraries mentioned in this page.

Modifying a Thrift RPC definition

Alluxio uses Thrift 0.9.3 for RPC communication between clients and servers. The .thrift files defined in core/common/src/thrift/ are used to auto-generate Java code for calling the RPCs on clients and implementing the RPCs on servers. To change a Thrift definition, you must first install the Thrift compiler. If you have brew, you can do this by running

brew install thrift@0.9
brew link --force thrift@0.9

Then to regenerate the Java code, run

bin/alluxio thriftGen

Modifying a Protocol Buffer Message

Alluxio uses Protocol Buffers 2.5.0 to read and write journal messages. The .proto files defined in core/protobuf/src/proto/ are used to auto-generate Java definitions for the protocol buffer messages. To change one of these messages, first read about updating a message type to make sure your change will not break backwards compatibility. Next, install protoc. If you have brew, you can do this by running

brew install protobuf@2.5
brew link --force protobuf@2.5

Then to regenerate the Java code, run

bin/alluxio protoGen

Usage of ./bin/alluxio

Most commands in bin/alluxio are for developers. The following table explains the description and the syntax of each command.

CommandArgsDescription
extensions None Manage extensions to core Alluxio
format [-s] Format Alluxio Master and all Workers. The option [-s] indicates that the command should only format when the underfs is local and doesn't already exist.
formatMaster None Format Alluxio master.
formatWorker None Format Alluxio Worker storage.
bootstrapConf <ALLUXIO_MASTER_HOSTNAME> Generate the bootstrap config file alluxio-env.sh with the specified ALLUXIO_MASTER_HOSTNAME, if the config file doesn't exist.
fs [fs-commands] Interact with Alluxio in command line style for basic file system operations. See [Command Line]({{ '/en/basic/Command-Line-Interface.html' | relativize_url }}) for more information.
fsadmin [fsadmin-commands] Command line tool for use by Alluxio filesystem admins. See [Fsadmin Command Line]({{ '/en/basic/operation/Admin-CLI.md' | relativize_url }}) for more information.
getConf [key] Look up a configuration key, or print all configuration.
loadufs <AlluxioPath> <UfsPath> [ExcludePathPrefixes] Loads files under UfsPath to the given AlluxioPath. This command has been deprecated since version 1.1. Use the alluxio fs mount <AlluxioURI> <UfsURI> command followed by the alluxio fs ls -R <AlluxioURI> command instead.
logLevel [--level <arg>] --logName <arg> [--target <arg>] Dynamically changes the log level of a class on the running Alluxio server. Try alluxio logLevel -help to see the meaning of each argument in detail.
runTest <Example> <ReadType> <WriteType> Run an end-to-end test on a Alluxio cluster. Example should be "Basic" or "BasicNonByteBuffer". ReadType should be "CACHE_PROMOTE", "CACHE", or "NO_CACHE". WriteType should be "MUST_CACHE", "CACHE_THROUGH" or "THROUGH".
runTests None Run all end-to-end tests on a Alluxio cluster. That is, execute the runTest command with all the possible args.
journalCrashTest [-creates <arg>] [-deletes <arg>] [-renames <arg>] [-maxAlive <arg>] [-testDir <arg>] [-totalTime <arg>] [-help] Test the Master Journal System in a crash scenario. Try alluxio journalCrashTest -help to see the meanings of each argument in detail, or you can run it without args by default.
readJournal [-help] [-noTimeout] Read a Alluxio journal file from stdin and write a human-readable version of it to stdout. You can run this on the journal file as alluxio readJournal < journal/FileSystemMaster/log.out.
upgradeJournal None Upgrade an Alluxio journal version 0 (Alluxio version < 1.5.0) to an Alluxio journal version 1 (Alluxio version >= 1.5.0).
killAll <WORD> Kill processes whose pid or command contains the WORD specified by the user.
copyDir <PATH> Copy the PATH to all worker nodes.
clearCache None Clear OS buffer cache of the machine. This command needs the root permission.
confDocGen None Generate configuration docs.
thriftGen None Generate all thrift code. See Change a Thrift RPC definition.
protoGen None Generate all protocol buffer code. See Change a Protocol Buffer Message.
version None Print Alluxio version.
validateConf None Validate Alluxio configuration.
validateEnv <TARGET> [NAME] Validate environment for Alluxio. TARGET can be one of the following values:
local run all validation tasks on local
master run master validation tasks on local
worker run worker validation tasks on local
all run corresponding validation tasks on all master nodes and worker nodes
masters run master validation tasks on all master nodes
workers run worker validation tasks on all worker nodes
NAME can be any task full name or prefix.
When NAME is given, only tasks with name starts with the prefix will run. For example, specifying NAME "master" or "ma" will run both tasks named master.rpc.port.available and master.web.port.available but not worker.rpc.port.available.
If NAME is not given, all tasks for the given TARGET will run.

In addition, these commands have different prerequisites. The prerequisite for the format, formatWorker, journalCrashTest, readJournal, version, validateConf and validateEnv commands is that you have already built Alluxio (see Build Alluxio Master Branch about how to build Alluxio manually). Further, the prerequisite for the fs, loadufs, logLevel, runTest and runTests commands is that you have a running Alluxio system.