Policy-Driven Data Management
- Mounting Multiple Storage Systems with Union UFS
- Reading and Writing Data on Union UFS
- Moving Data with Policies
- Managing Policies
- Example: Running Alluxio Locally with Union UFS
Moving data from one storage system to another, while keeping computation jobs running correctly is a common challenge for many Alluxio users. Traditionally this is done by manually copying the data to the destination, asking every user of the data to update their applications and queries to use the new URI, waiting until all the updates are complete, then finally removing the original copy. With Alluxio, this can be done seamlessly by mounting multiple storage systems to a single mount point using a union UFS.
For example, if a user wants to move data from
s3://user_bucket/, they can mount both
locations using a union-UFS to Alluxio under path
/union/, and create a migration policy to move data from HDFS to S3.
Alluxio will enforce the policy and automatically move the data, while users can keep using the same Alluxio path
alluxio://host/union to access their data, without worrying that their computation job will fail after the data is moved.
On a union UFS, each under storage system is mounted as a sub-UFS. A path in Alluxio is mapped to one or more corresponding files in the mounted sub-UFSes. When a file is written to Alluxio, it can be configured to be written to one or more sub-UFSes. When a file is read from Alluxio, it can be read transparently from any sub-UFS mounted with union UFS. This allows user to move data among storage systems easily while accessing data with a single Alluxio URI regardless which storage system the file is in.
This page describes the instructions to configure a union UFS to mount multiple storage systems, and setup policies to move data among them.
Mounting Multiple Storage Systems with Union UFS
Note: Before attempting a union UFS mount, it will help to validate that each sub-UFS is able to perform basic
operations on its own by using the
./bin/alluxio runTests [--directory <PATH>] command.
Start by mounting each sub-UFS as its own mount point and making sure it passes the tests.
Once that is validated for all the different sub-UFSs, make sure they are unmounted from Alluxio before proceeding with
configuring the union UFS.
Similar to mounting any other under storage system, a union UFS can be mounted using
alluxio fs mount command:
./bin/alluxio fs mount \ --option alluxio-union.<UFS_A_ALIAS>.uri=<UFS_A_URI> \ --option alluxio-union.<UFS_A_ALIAS>.option.<KEY>=<VALUE> \ --option alluxio-union.<UFS_B_ALIAS>.uri=<UFS_B_URI> \ --option alluxio-union.<UFS_B_ALIAS>.option.<KEY>=<VALUE> \ --option alluxio-union.priority.read=<ALIAS_1>,<ALIAS_2>,<ALIAS_3>... \ --option alluxio-union.collection.create=<UFS_ALIASES> \ <ALLUXIO_MOUNT_PATH> union://<UNION_UFS_AUTHORITY>/
In the command above:
alluxio-union.<UFS_ALIAS>.urispecifies a URI of a sub under storage system(sub-UFS) to be mounted.
<UFS_ALIAS>should be replaced with an alias representing the sub-UFS. Users can specify multiple UFSs to be mounted with different aliases. For example,
alluxio-union.hdfs.uri=hdfs://local_hdfs/user_a/tells the union UFS to mount a sub-UFS at
hdfs://local_hdfs/user_a/with the alias
alluxio-union.<UFS_ALIAS>.option.<KEY>specifies an optional UFS option for a specific sub-UFS.
<UFS_ALIAS>denotes an alias of sub-UFS defined by
<KEY>denotes the option key to be set to the target sub-UFS. For example,
aws.accessKeyIdoption for sub-UFS
alluxio-union.priority.readspecifies an ordered list of UFS aliases to set the read priority of sub-UFSes. When a file is read from a union UFS, it will be attempted on sub-UFSes in the same order as the aliases in this list. The first sub-UFS with the file available for reading will be used. All sub-UFS aliases must appear in this list.
alluxio-union.collection.createspecifies an unordered list of UFSes to write new files to. When a new file is written to a union UFS, it will be written to all the UFSes whose aliases specified in this list.
<ALLUXIO_MOUNT_PATH>specifies the Alluxio path where the union UFS is mounted.
<UNION_UFS_AUTHORITY>specifies a unique authority of the mounted union UFS. It can be empty or any arbitrary name. This is used to identify a specific union UFS.
For example, the following command can be used to mount an HDFS directory and an S3 bucket under the Alluxio path
./bin/alluxio fs mount \ --option alluxio-union.hdfs.uri=hdfs://local_hdfs/user_a/ \ --option alluxio-union.hdfs.option.alluxio.underfs.hdfs.configuration=/opt/hdfs/core-site.xml:/opt/hdfs/hdfs-site.xml \ --option alluxio-union.s3.uri=s3://mybucket/ \ --option alluxio-union.s3.option.aws.accessKeyId=MYKEYID \ --option alluxio-union.s3.option.aws.secretKey=MYSECRETKEY \ --option alluxio-union.priority.read=hdfs,s3 \ --option alluxio-union.collection.create=s3 \ /union union://union_ufs_1/
Reading and Writing Data on Union UFS
Data on a union UFS can be accessed just like any other UFS. Files that exist in any of the sub-UFSes will show up in the
mount point. For example, if sub-UFS
a contains a file at path
/a/b and another one at
/c, and sub-UFS
a file at path
/a/d and and a file at
/c, then the union UFS will expose a directory structure as follows:
+-a | +b | +d | +-c
When file data is read from a union UFS, it will be read from the sub-UFS of the highest priority which contains the file.
In the above example, if sub-UFS
a has the highest priority, then reading data from
/c will end up
reading from sub-UFS
a. Reading data from
/a/d will end up reading from sub-UFS
b since only
b has the file.
When file data is written to a union UFS, it will be written to all sub-UFSes specified in the
alluxio-union.collection.create property. A write request will complete after the file is completed on all target
sub-UFSes. If one of the sub-UFSes fails to write the file, the write request will fail.
It may be helpful to use the
./bin/alluxio runTests [--directory <PATH>] command to validate that most of the basic read/write
operations will work on the union UFS.
Moving Data with Policies
Files in a union UFS can be moved between sub-UFSes by setting policies. A policy can be added by the following command:
./bin/alluxio fs policy add <PATH> "[<POLICY_NAME>:]ufsMigrate(<TIME_CONDITION>, <LOCATION>:<OPERATION>, ...)"
In the command above:
<POLICY_NAME>specifies the name of the policy, must be a unique string containing only alphanumeric characters. If the name is not specified, an automatically generated name will be used.
<PATH>specifies an Alluxio path where the policy will apply.
<TIME_CONDITION>specifies a time condition that the policy will be executed. Time condition is specified in the format of
conditionType(startTime[, endTime])There are three types of time condition:
olderThan: the policy will be executed when a file or directory is older than a certain period of time.
unusedFor: the policy will be executed when a file or directory is not used for a certain period of time.
unmodifiedFor: the policy will be executed when a file or directory is not modified for a certain period of time.
dateFromFileNameOlderThan: the date will be extracted from the file name based on the pattern specified by the user. The pattern needs to include
YYYY, which represents the year,
MM, which represents the month,
DD, which represents the date of the month. For example, if the pattern is
bbbbYYYYMMDD, then we can extract date from some file names like:
aaaa2023031aaaa. An exception will be thrown when users are trying to add a pattern without
DD. You can add the policy with a flag named
file_date_patternto specify the pattern. For example,
/bin/alluxio fs policy add /union/test/ "ufsMigrate(dateFromFileNameOlderThan(2d), UFS[a]:REMOVE, UFS[b]:STORE)" --file_date_pattern YYYY-MM-DD
The time period can be specified using a single start time, or a start time and an end time. For each time value, use
dto indicate time unit seconds, minutes, hours, and days. For example,
unmodifiedFor(30m)sets a policy to execute after a file is not modified for 30 minutes.
olderThan(3h, 1d)sets a policy to execute after a file is older than 3 hours but younger than one day.
unusedFor(7d)sets a policy to execute after a file is not used for 7 days.
dateFromFileNameOlderThan(2d)sets a policy to execute after the date from the file name is older than 2 days.
<LOCATION>:<OPERATION>specifies a file operation to be executed.
<LOCATION>specifies the target storage of the operation. Use
UFS[<UFS_ALIAS>]to refer to a sub-UFS.
<OPERATION>specifies how the data is affected on this storage location. It can be either
STOREto indicate the file data should be stored on this location, or
REMOVEto indicate that data should be removed from this location.
For example, if we have a union UFS mount point at Alluxio mount point
/union/ with the following options:
the command below will add a policy to move files under the Alluxio path
/union/mydir from the corresponding
hdfs://local_hdfs/user_a/mydir to the S3 path
s3://mybucket/mydir once they are older than 2 days:
./bin/alluxio fs policy add /union/mydir "ufsMigrate(olderThan(2d), UFS[hdfs]:REMOVE, UFS[s3]:STORE)"
After a policy is added, it will be executed in the background after all its requirements are met. Users can always access
files with their Alluxio URIs regardless whether they are migrated with a policy or not. A policy can also be added to
a mount point using the
--policy <POLICY_DESCRIPTION> option in
alluxio fs mount command, for example:
./bin/alluxio fs mount \ --option alluxio-union.a.uri=/tmp/ufs1/ \ --option alluxio-union.b.uri=/tmp/ufs2/ \ --option alluxio-union.priority.read=a,b \ --option alluxio-union.collection.create=a \ --policy "ufsMigrate(olderThan(2d), UFS[a]:REMOVE, UFS[b]:STORE)" \ /union/ union://test/
will mount a union UFS and add a policy to migrate files inside the mount point at
To check the status of all the policies, use the following command:
./bin/alluxio fsadmin report policy"
This command will print out a summary of all the policy execution status like the following:
Total Policy Number: 2 Policy Name: ufsMigrate-/union Last scanned at 03-31-2022 23:22:41:700 Policy Name: ufsStore-/union Last scanned at 03-31-2022 23:22:41:800 Files/Directories Scanned: 26 Files/Directories Pending Action: 5 Files/Directories Completed: 21 Execution status since server started: Actions Running: 12 Actions Succeeded: 37 2 Most recent errors since server started: ActionId: 30be56b2-8eae-46c8-a22a-474e8168320d Description: DATA(UFS[b]:STORE, UFS[a]:REMOVE) on file(/union/job_master.log) with id(754974719) ErrorMessage: 04-11-2022 12:50:52:450 - Job failed: Task execution failed: Union InputStream failed to open file at union://test/job_master.log ActionId: 69e34fb9-eb73-4abe-89d2-11960839b9a6 Description: DATA(UFS[b]:STORE, UFS[a]:REMOVE) on file(/union/job_worker.log) with id(905969663) ErrorMessage: 04-11-2022 12:50:52:458 - Job failed: Task execution failed: Union InputStream failed to open file at union://test/job_worker.log
In the output:
Files/Directories Scannedrepresents the number of files/directories evaluated for this policy during the last scan.
Files/Directories Pending Actionrepresents the number of files/directories that have scheduled actions to be executed per definition of the policy.
Files/Directories Completedrepresents the number of files/directories whose statuses match the completion state of the policy.
Actions Runningrepresents the number of actions that are currently being executed to enforce the policy.
Actions Succeededrepresents the number of actions that are completed successfully.
ActionIdrepresents the unique UUID of the action.
Descriptionrepresents the description of what action is doing.
ErrorMessagerepresents the error message of the failed action.
To list all policies, use the following command:
./bin/alluxio fs policy list"
To check the status of a specific policy, use the following command:
./bin/alluxio fs policy status <POLICY_NAME>"
<POLICY_NAME> should be the name of the policy that is listed in the
policy list command or
fsadmin report policy command. This command will
print out a summary of the policy execution status like the following:
Policy <POLICY_NAME>: Last scanned at 11-01-2019 12:23:01:012 Files/Directories Scanned: 26 Files/Directories Pending Action: 5 Files/Directories Completed: 21 Execution status since server started: Actions Running: 12 Actions Succeeded: 37
To check the details of a specific action in the policy, locate the action id from either
policy status command or
fsadmin report policy command, and run:
bin/alluxio fs policy status <POLICY_NAME> --action <ACTION_ID>
Alternatively, the action details can also be fetched by the unionFS file path the action operated on:
bin/alluxio fs policy status <POLICY_NAME> --path <FILE_PATH>
You should only input one of these two options to specify the information you want otherwise actionId would be used to find the action information. We only record finished actions which action status would be either COMMITTED or FAILED.
This command will print out the action status like the following:
Related Actions: ActionId:1648793869143 FilePath: /union/EULA Timestamp: 03-31-2022 23:20:01:614 Description: DATA(UFS[b]:STORE, UFS[a]:REMOVE) Status: COMMITTED
In the output:
ActionIdrepresents the unique UUID of the action.
FilePathrepresents the unionFS file path
Timestamprepresents the last time stamp action get recorded.
Descriptionrepresents the description of what action is doing.
Statusrepresents the status of action.
To remove a policy so it is no longer executed, use the following command:
./bin/alluxio fs policy remove <POLICY_NAME>"
Alluxio master periodically scans file system for policies that can be executed. The default scan interval is 24 hours.
It can be changed by setting
conf/alluxio-site.properties. Please avoid setting
a short scan interval on a large file system as it can be resource intensive to scan a large amount of files.
You can also manually trigger the scan on all the policies with the following command.
./bin/alluxio fs policy trigger
To completely disable policy execution, set
Example: Running Alluxio Locally with Union UFS
First, start the Alluxio servers locally:
./bin/alluxio format ./bin/alluxio-start.sh local
If your ramdisk is not mounted, likely because this is the first time you are running Alluxio, you may need to start
Alluxio with the
./bin/alluxio-start.sh local SudoMount
This will start one Alluxio master and one Alluxio worker locally. You can see the master UI at http://localhost:19999.
Run a simple example program:
If the test fails with permission errors, make sure that the current user (
read/write access to the local directory mounted to Alluxio. By default,
the login user is the current user of the host OS. To change the user, set the value of
conf/alluxio-site.properties to the desired username.
After this succeeds, create two local directories to be used as sub-UFSes. In this example we will use
mkdir /tmp/ufs1 mkdir /tmp/ufs2
Mount the union UFS with the local sub-UFSes using the following command:
./bin/alluxio fs mount \ --option alluxio-union.a.uri=/tmp/ufs1/ \ --option alluxio-union.b.uri=/tmp/ufs2/ \ --option alluxio-union.priority.read=a,b \ --option alluxio-union.collection.create=a \ /union/ union://test/
Setup a policy to move from UFS
a to UFS
./bin/alluxio fs policy add /union/ "ufsMigrate(olderThan(1m), UFS[a]:REMOVE, UFS[b]:STORE)"
To test the mounted union UFS, run the following command to copy a new file to the mount point and persist it to UFS:
./bin/alluxio fs -Dalluxio.user.file.writetype.default=CACHE_THROUGH copyFromLocal LICENSE /union
After the command finishes, you should find the
LICENSE file in local directory
/tmp/ufs1, but not in
Wait a minute for the policy to be executed, then check the local directories again. The file should now be moved from
Stop Alluxio by running: