Alluxio Standalone on a Cluster

Slack Docker Pulls GitHub edit source

Download Alluxio

First download the Alluxio tar file, and extract it.

tar xvfz alluxio-1.4.0-bin.tar.gz

Configure Alluxio

Using the bootstrapConf argument to the bin/alluxio script

The alluxio script also contains logic to create a basic config for a cluster. If you run:

cd alluxio
./bin/alluxio bootstrapConf <alluxio_master_hostname>

and there is no existing conf/ file, then the script will create one with the appropriate settings for a cluster with a master node running at <alluxio_master_hostname>.

This script needs to be run on each node you wish to configure.

The script will configure your workers to use 2/3 of the total memory on each worker. This amount can be changed by editing the created conf/ file on the worker.

Using script

There is another way to create file instead of using bootstrapConf command. In the ${ALLUXIO_HOME}/conf directory, copy to Update ALLUXIO_MASTER_HOSTNAME to the hostname of the machine you plan to run Alluxio Master on. Add the IP addresses of all the worker nodes to the conf/workers file. Finally, sync all the information to worker nodes. You can use

./bin/alluxio copyDir <dirname>

to sync files and folders to all hosts specified in the alluxio/conf/workers file.

Start Alluxio

Now, you can start Alluxio:

cd alluxio
./bin/alluxio format
./bin/ # use the right parameters here. e.g. all Mount
# Notice: the Mount and SudoMount parameters will format the existing RamFS.

To verify that Alluxio is running, you can visit http://<alluxio_master_hostname>:19999, check the log in the directory alluxio/logs, or run a sample program:

./bin/alluxio runTests

Note: If you are using EC2, make sure the security group settings on the master node allows incoming connections on the alluxio web UI port.