Journal

Slack Docker Pulls

Alluxio maintains a journal to support persistence of metadata operations. When a request modifies Alluxio state, e.g. creating or renaming files, the Alluxio master will write a journal entry for the operation before returning a successful response to the client. The journal entry is written to a persistent storage such as disk or HDFS, so even if the Alluxio master process is killed, state will be recovered on restart. Alluxio supports two types of journals: Under File Storage (UFS) Journal and Embedded Journal.

Comparison

The UFS journal simplifies certain aspects of Alluxio operation, but it relies on an external Zookeeper cluster for coordination, and relies on a UFS for persistent storage. To get reasonable performance, the UFS journal requires a UFS that supports fast streaming writes, such as HDFS or NFS.

The embedded journal does its own coordination and persistent storage, but has a few limitations.

First, n masters using the embedded journal can tolerate only floor(n/2) master failures, compared to n-1 for UFS journal. For example, With 3 masters, UFS journal can tolerate 2 failures, while embedded journal can only tolerate 1. However, UFS journal depends on Zookeeper, which similarly only supports floor(#zookeeper_nodes / 2) failures.

The other limitation is that embedded journal does not support dynamically changing master membership. With UFS journal, replacing a master on host1 with a new master on host2 is as simple as starting a new master on host2, then killing the master on host1. Changing the masters in an embedded journal cluster requires backing up the cluster, shutting it down, and then starting up again with the new masters using the backup.

If a fast UFS and Zookeeper cluster are readily available and stable, it is recommended to use the UFS journal. Otherwise, we recommend using the embedded journal.

UFS Journal Configuration

The most important configuration value to set for the journal is alluxio.master.journal.folder. This must be set to a shared filesystem available to all masters. In single-master mode, a local filesystem path is fine. With multiple masters distributed across different machines, the shared folder should be in a distributed system that supports flush such as HDFS or NFS. It is not recommended to put the journal in an object store. With an object store, every update to the journal requires a new object to be created, which is prohibitively slow for most serious use cases.

The journal folder must be configured on all masters nodes.

Configuration examples:

alluxio.master.journal.folder=hdfs://[namenodeserver]:[namenodeport]/dir/alluxio_journal
alluxio.master.journal.folder=/opt/alluxio/journal

Zookeeper configuration must be configured on all masters, workers, and clients.

alluxio.zookeeper.enabled=true
alluxio.zookeeper.address=zkhost1:2181,zkhost2:2181,zkhost3:2181

Embedded Journal Configuration

Required configuration

The configuration specified below should be applied to both Alluxio servers and Alluxio clients.

Set the journal type to “EMBEDDED”

alluxio.master.journal.type=EMBEDDED

Set the addresses of all masters in the cluster. The default embedded journal port is 19200.

alluxio.master.embedded.journal.addresses=master_hostname_1:19200,master_hostname_2:19200,master_hostname_3:19200

Optional configuration

  • alluxio.master.embedded.journal.port: The port masters use for embedded journal communication. Default: 19200.
  • alluxio.master.port: The port masters use for RPCs. Default: 19998.
  • alluxio.master.rpc.addresses: A list of comma-separated host:port RPC addresses where the client should look for masters when using multiple masters without Zookeeper. This property is not used when Zookeeper is enabled, since Zookeeper already stores the master addresses. If this is not set, clients will look for masters using the hostnames from alluxio.master.embedded.journal.addresses and the master rpc port.

Job service configuration

It is usually best not to set any of these - by default the job master will use the same hostnames as the Alluxio master, so it is enough to set only alluxio.master.embedded.journal.addresses. These properties only need to be set when the job service is being run independent from the rest of the system or using a non-standard port.

  • alluxio.job.master.embedded.journal.port: the port job masters use for embedded journal communications. Default: 20003.
  • alluxio.job.master.embedded.journal.addresses: a comma-separated list of journal addresses for all job masters in the cluster. The format is ‘hostname1:port1,hostname2:port2,…’.
  • alluxio.job.master.rpc.addresses: A list of comma-separated host:port RPC addresses where the client should look for job masters when using multiple job masters without Zookeeper. This property is not used when Zookeeper is enabled, since Zookeeper already stores the job master addresses. If this is not set, clients will look for job masters using the hostnames from alluxio.master.embedded.journal.addresses and the job master rpc port.

Formatting

Before starting Alluxio for the first time, the journal must be formatted.

WARNING: formatting the journal will delete all Alluxio metadata

$ bin/alluxio formatMasters

When using the UFS journal, this command only needs to be run from one master. Embedded journal requires running format on each master. If embedded journal is configured, formatMasters will attempt to use conf/masters to ssh to each master and format it. If ssh is not possible or conf/masters is not configured, you will need to manually run formatMasters on each master.

Backup

Alluxio supports taking journal backups so that Alluxio metadata can be restored to a previous point in time. Generating a backup causes temporary service unavailability while the backup is happening.

To generate a backup, use the fsadmin backup CLI command.

$ bin/alluxio fsadmin backup

By default, this will write a backup named alluxio-journal-YYYY-MM-DD-timestamp.gz to the “/alluxio_backups” directory of the root under file system, e.g. hdfs://cluster/alluxio_backups. This default backup directory can be configured by setting alluxio.master.backup.directory

alluxio.master.backup.directory=/alluxio/backups

See the backup command documentation for additional options for specifying where to write a backup.

Restore

To restore the Alluxio system from a journal backup, stop the system, format the journal, then restart the system, passing the URI of the backup with the -i (import) flag. If conf/masters is configured to manage the cluster from a single node, run

$ bin/alluxio-stop.sh masters
$ bin/alluxio formatMasters
$ bin/alluxio-start.sh -i <backup_uri> masters

If not using conf/masters to manage the cluster, first stop all masters indivdually, then format, then start all masters individually. UFS journal should be formatted from a single master, while embedded journal must be formatted on all masters.

The <backup_uri> should be a full URI path that is available to all masters, e.g. hdfs://[namenodeserver]:[namenodeport]/alluxio_backups/alluxio-journal-YYYY-MM-DD-timestamp.gz

If the restore succeeds, you should see a log message along the lines of

INFO AlluxioMasterProcess - Restored 57 entries from backup

in the primary master logs.