- Formatting the journal
Alluxio is fault-tolerant: force-killing the system will not lose metadata. To achieve this, the master writes edit logs of all metadata changes. On startup, a recovering master will read the edit logs to restore itself back to its previous state. We use the term “journal” to refer to the system of edit logs used to support fault-tolerance. The purpose of this documentation is to help Alluxio administrators understand and manage the Alluxio journal.
The most important configuration value to set for the journal is
alluxio.master.journal.folder. This must be set to a filesystem folder that is
available to all masters. In single-master mode, a local filesystem path may be
used for simplicity. With
multiple masters distributed across different machines, the folder must
be in a distributed system where all masters can access it. The journal folder
should be in a filesystem that supports flush such as HDFS or NFS. It is not
recommended to put the journal in an object store like S3. With an object store, every
metadata operation requires a new object to be created, which is
prohibitively slow for most serious use cases.
Use HDFS to store the journal:
Use the local file system to store the journal:
Formatting the journal
Formatting the journal deletes all of its content and restores it to a fresh state. Before starting Alluxio for the first time, the journal must be formatted.
# This permanently deletes all Alluxio metadata, so be careful with this operation bin/alluxio formatMaster
Manually backing up the journal
Alluxio supports taking journal backups so that Alluxio metadata can be restored to a previous point in time. Generating a backup causes temporary service unavailability while the backup is written.
To generate a backup, use the
fsadmin backup CLI command.
bin/alluxio fsadmin backup
By default, this will write a backup named
alluxio-journal-YYYY-MM-DD-timestamp.gz to the “/alluxio_backups” directory of
the root under storage system, e.g. hdfs://cluster/alluxio_backups. This default
backup directory can be configured by setting
See the backup command documentation for additional backup options.
Automatically backing up the journal
Alluxio supports automatically taking primary master metadata snapshots every day at a fixed time
so that Alluxio metadata can be restored to at most one day before.
This functionality is enabled by setting the following property in
The time to take daily snapshots is defined by
alluxio.master.daily.backup.time. For example, if
a user specified
alluxio.master.daily.backup.time=05:30, the Alluxio primary master will back up its metadata
alluxio.master.backup.directory of the root UFS every day at 5:30am UTC.
We recommend setting the backup time to an off-peak time to avoid interfering with other users of the system.
In the daily backup, the backup directory needs to be an absolute path within the root UFS.
For example, if
the default backup directory would be
The files to retain in the backup directory is limited by
Users can set this property to the number of backup files they want to keep in the backup directory.
Restoring from a backup
To restore the Alluxio system from a journal backup, stop the system, format the
journal, then restart the system, passing the URI of the backup with the
bin/alluxio-stop.sh masters bin/alluxio formatMaster bin/alluxio-start.sh -i <backup_uri> masters
<backup_uri> should be a full URI path that is available to all masters, e.g.
If starting up masters individually, pass the
-i argument to each one. The master which
becomes leader first will import the journal backup, and the rest will ignore the
If the restore succeeds, you should see a log message along the lines of
INFO AlluxioMasterProcess - Restored 57 entries from backup
in the leading master logs.
If the journal is stored in a shared storage system like HDFS, changing masters is easy.
As long as the new master sets
alluxio.master.journal.folder the same as the old
master, it will start up in the same state that the old master left off.
If the journal is stored on the local filesystem, the journal folder needs to be copied to the new master.
Managing the journal size
When running with a single master, the journal folder size will grow indefinitely as metadata operations are written to journal log files. To address this, production deployments should run in HA mode with multiple Alluxio masters. The standby masters will create checkpoints of the master state and clean up the logs that were written prior to the checkpoints. For example, if 3 million Alluxio files were created and then 2 million were deleted, the journal logs would contain 5 million total entries. Then if a checkpoint is taken, the checkpoint will contain only the metadata for the 1 million remaining files, and the original 5 million entries will be deleted.
By default, checkpoints are automatically taken every 2 million entries. This can be configured by
alluxio.master.journal.checkpoint.period.entries on the masters. Setting
the value lower will reduce the amount of disk space needed by the journal at the
cost of additional work for the standby masters.
If HA mode is not an option, it is possible to run a master on the same node as a dedicated standby master. This second master exists only to write checkpoints, and will not serve client requests if the leading master dies. In this setup, both masters have similar memory requirements since they both need to hold all Alluxio metadata in memory. To start a dedicated standby master for writing periodic checkpoints, run
Recovering from journal issues
The journal is integral to Alluxio’s health. If the filesystem storing the journal loses availability, no metadata operations can be performed on Alluxio. Similarly, if the journal is accidentally deleted or its storage system becomes corrupted, Alluxio must be reformatted to recover. To avoid the need for full reformatting, we recommend taking regular journal backups at a time when the cluster is under low load. Then if something happens to the journal, you can recover from one of the backups.