- UFS Journal Configuration
- Embedded Journal Configuration
Alluxio maintains a journal to support persistence of metadata operations. When a request modifies Alluxio state, e.g. creating or renaming files, the Alluxio master will write a journal entry for the operation before returning a successful response to the client. The journal entry is written to a persistent storage such as disk or HDFS, so even if the Alluxio master process is killed, state will be recovered on restart. Alluxio supports two types of journals: Under File Storage (UFS) Journal and Embedded Journal.
The UFS journal simplifies certain aspects of Alluxio operation, but it relies on an external Zookeeper cluster for coordination, and relies on a UFS for persistent storage. To get reasonable performance, the UFS journal requires a UFS that supports fast streaming writes, such as HDFS or NFS.
The embedded journal does its own coordination and persistent storage, but has a few limitations.
n masters using the embedded journal can tolerate only
master failures, compared to
n-1 for UFS journal.
For example, With
3 masters, UFS journal can tolerate
2 failures, while embedded journal can only tolerate
However, UFS journal depends on Zookeeper, which similarly only supports
floor(#zookeeper_nodes / 2) failures.
The other limitation is that embedded journal does not support dynamically changing master membership.
With UFS journal, replacing a master on
host1 with a new master on
host2 is as simple as starting a new
host2, then killing the master on
host1. Changing the masters in an embedded journal cluster
requires backing up the cluster, shutting it down, and then starting up again with the new masters
using the backup.
If a fast UFS and Zookeeper cluster are readily available and stable, it is recommended to use the UFS journal. Otherwise, we recommend using the embedded journal.
UFS Journal Configuration
The most important configuration value to set for the journal is
alluxio.master.journal.folder. This must be set to a shared filesystem available
to all masters. In single-master mode, a local filesystem path is fine. With
multiple masters distributed across different machines, the shared folder should
be in a distributed system that supports flush such as HDFS or NFS. It is not
recommended to put the journal in an object store. With an object store, every
update to the journal requires a new object to be created, which is
prohibitively slow for most serious use cases.
The journal folder must be configured on all masters nodes.
Zookeeper configuration must be configured on all masters, workers, and clients.
Embedded Journal Configuration
The configuration specified below should be applied to both Alluxio servers and Alluxio clients.
Set the journal type to “EMBEDDED”
Set the addresses of all masters in the cluster. The default embedded journal port
alluxio.master.embedded.journal.port: The port masters use for embedded journal communication. Default:
alluxio.master.port: The port masters use for RPCs. Default:
alluxio.master.rpc.addresses: A list of comma-separated host:port RPC addresses where the client should look for masters when using multiple masters without Zookeeper. This property is not used when Zookeeper is enabled, since Zookeeper already stores the master addresses. If this is not set, clients will look for masters using the hostnames from
alluxio.master.embedded.journal.addressesand the master rpc port.
Job service configuration
It is usually best not to set any of these - by default the job master will use the
same hostnames as the Alluxio master, so it is enough to set only
These properties only need to be set when the job service is being run independent from the rest of the system
or using a non-standard port.
alluxio.job.master.embedded.journal.port: the port job masters use for embedded journal communications. Default:
alluxio.job.master.embedded.journal.addresses: a comma-separated list of journal addresses for all job masters in the cluster. The format is ‘hostname1:port1,hostname2:port2,…’.
alluxio.job.master.rpc.addresses: A list of comma-separated host:port RPC addresses where the client should look for job masters when using multiple job masters without Zookeeper. This property is not used when Zookeeper is enabled, since Zookeeper already stores the job master addresses. If this is not set, clients will look for job masters using the hostnames from
alluxio.master.embedded.journal.addressesand the job master rpc port.
Before starting Alluxio for the first time, the journal must be formatted.
WARNING: formatting the journal will delete all Alluxio metadata
$ bin/alluxio formatMasters
When using the UFS journal, this command only needs to be run from one master.
Embedded journal requires running format on each master.
If embedded journal is configured, formatMasters will attempt to use
conf/masters to ssh to each
master and format it. If ssh is not possible or
conf/masters is not configured, you will need to
formatMasters on each master.
Alluxio supports taking journal backups so that Alluxio metadata can be restored to a previous point in time. Generating a backup causes temporary service unavailability while the backup is happening.
To generate a backup, use the
fsadmin backup CLI command.
$ bin/alluxio fsadmin backup
By default, this will write a backup named
alluxio-journal-YYYY-MM-DD-timestamp.gz to the “/alluxio_backups” directory of
the root under file system, e.g.
hdfs://cluster/alluxio_backups. This default
backup directory can be configured by setting
See the backup command documentation for additional options for specifying where to write a backup.
To restore the Alluxio system from a journal backup, stop the system, format the
journal, then restart the system, passing the URI of the backup with the
(import) flag. If
conf/masters is configured to manage the cluster from a single node, run
$ bin/alluxio-stop.sh masters $ bin/alluxio formatMasters $ bin/alluxio-start.sh -i <backup_uri> masters
If not using
conf/masters to manage the cluster, first stop all masters
indivdually, then format, then start all masters individually. UFS journal should be formatted from
a single master, while embedded journal must be formatted on all masters.
<backup_uri> should be a full URI path that is available to all masters, e.g.
If the restore succeeds, you should see a log message along the lines of
INFO AlluxioMasterProcess - Restored 57 entries from backup
in the primary master logs.