User Command Line Interface
Alluxio’s command line interface provides user access to various operations, such as:
- Start or stop processes
- Filesystem operations
- Administrative commands
Invoke the executable to view the possible subcommands:
$ ./bin/alluxio
Usage:
bin/alluxio [command]
Available Commands:
conf Get, set, and validate configuration settings, primarily those defined in conf/alluxio-site.properties
exec Run the main method of an Alluxio class, or end-to-end tests on an Alluxio cluster.
fs Operations to interface with the Alluxio filesystem
help Help about any command
info Retrieve and/or display info about the running Alluxio cluster
job Command line tool for interacting with the job service.
journal Journal related operations
license Check and manage license status
mount Operations to manage mount points
process Start/stop cluster processes or remove workers
quota Operations to manage quota definitions
security Security server related commands
Flags:
--debug-log True to enable debug logging
--skip-license-warnings Skip the automatic warnings before expiration
Use "bin/alluxio [command] --help" for more information about a command.
To set JVM system properties as part of the command, set the -D
flag in the form of -Dproperty=value
.
To attach debugging java options specified by $ALLUXIO_USER_ATTACH_OPTS
, set the --attach-debug
flag
Note that, as a part of Alluxio deployment, the Alluxio shell will also take the configuration in ${ALLUXIO_HOME}/conf/alluxio-site.properties
when it is run from Alluxio installation at ${ALLUXIO_HOME}
.
conf
Get, set, and validate configuration settings, primarily those defined in conf/alluxio-site.properties
conf get
Usage: bin/alluxio conf get [key] [flags]
The get command prints the configured value for the given key. If the key is invalid, it returns a nonzero exit code. If the key is valid but isn’t set, an empty string is printed. If no key is specified, the full configuration is printed.
Note: This command does not require the Alluxio cluster to be running.
Flags:
--master
: Show configuration properties used by the master (Default: false)--source
: Show source of the configuration property instead of the value (Default: false)--unit
: Unit of the value to return, converted to correspond to the given unit. E.g., with “–unit KB”, a configuration value of “4096B” will return 4 Possible options include B, KB, MB, GB, TP, PB, MS, S, M, H, D (Default: “”)
Examples:
# Display all the current node configuration
$ ./bin/alluxio conf get
# Display the value of a property key
$ ./bin/alluxio conf get alluxio.master.hostname
# Display the configuration of the current running Alluxio leading master
$ ./bin/alluxio conf get --master
# Display the source of the configuration
$ ./bin/alluxio conf get --source
# Display the values in a given unit
$ ./bin/alluxio conf get alluxio.user.block.size.bytes.default --unit KB
$ ./bin/alluxio conf get alluxio.master.journal.flush.timeout --unit S
exec
Run the main method of an Alluxio class, or end-to-end tests on an Alluxio cluster.
exec edgeTest
Usage: bin/alluxio exec edgeTest [flags]
Test whether the edge runs successfully.
Flags:
--no-cluster
: only interact with edge and ufs, not to access Dora cluster. (Default: false)--path
: (Required) The path can be:- An Alluxio path like ‘alluxio:///data’ only when not set ‘–no-cluster’ flag
- A path without scheme, like ‘/’ or ‘/s3’. This is a syntactic sugar for Alluxio paths.
- A UFS path with scheme, like ‘s3://bucket/data’ The path must exists in one of the mount points in Alluxio namespace.
exec ufsIOTest
Usage: bin/alluxio exec ufsIOTest [flags]
A benchmarking tool for the I/O between Alluxio and UFS. This test will measure the I/O throughput between Alluxio workers and the specified UFS path. Each worker will create concurrent clients to first generate test files of the specified size then read those files. The write/read I/O throughput will be measured in the process.
Flags:
--io-size
: specifies the amount of data each thread writes/reads. (Default: “”)--java-opt
: The java options to add to the command line to for the task. This can be repeated. The options must be quoted and prefixed with a space. For example: –java-opt “ -Xmx4g” –java-opt “ -Xms2g”. (Default: [])--path
: (Required) specifies the path to write/read temporary data in.--threads
: specifies the number of threads to concurrently use on each worker. (Default: 4)
Examples:
# This runs the I/O benchmark to HDFS in your process locally
$ ./bin/alluxio runUfsIOTest --path hdfs://<hdfs-address>
# This invokes the I/O benchmark to HDFS in the Alluxio worker
# 2 threads will be created on each worker
# Each thread is writing then reading 512m of data
$ ./bin/alluxio runUfsIOTest --path hdfs://<hdfs-address> --io-size 512m --threads 2
exec ufsTest
Usage: bin/alluxio exec ufsTest [flags]
Test the integration between Alluxio and the given UFS to validate UFS semantics
Flags:
--path
: (Required) the full UFS path to run tests against.--test
: Test name, this option can be passed multiple times to indicate multipleZ tests (Default: [])
fs
Operations to interface with the Alluxio filesystem For commands that take Alluxio URIs as an argument such as ls or mkdir, the argument should be either
- A complete Alluxio URI, such as alluxio://<masterHostname>:<masterPort>/<path>
- A path without its scheme header, such as /path, in order to use the default hostname and port set in alluxio-site.properties
Note: All fs commands require the Alluxio cluster to be running.
Most of the commands which require path components allow wildcard arguments for ease of use.
For example, the commandbin/alluxio fs rm '/data/2014*'
deletes anything in the data directory with a prefix of 2014.
Some shells will attempt to glob the input paths, causing strange errors.
As a workaround, you can disable globbing (depending on the shell type; for example, set -f) or by escaping wildcards
For example, the command bin/alluxio fs cat /\\*
uses the escape backslash character twice.
This is because the shell script will eventually call a java program which should have the final escaped parameters cat /\\*
.
fs cat
Usage: bin/alluxio fs cat [path]
The cat command prints the contents of a file in Alluxio to the shell.
Examples:
# Print the contents of /output/part-00000
$ ./bin/alluxio fs cat /output/part-00000
fs check-cached
Usage: bin/alluxio fs check-cached --path|--index-file <path> [--limit <limit-size>] [flags]
Checks if files under a path have been cached in alluxio.
Flags:
--index-file
: The index file on local that contains a list of file paths to check. Each line should contain a ufs file path. (Default: “”)--limit
: Limit number of files to check (Default: 1000)--path
: The path to check caching status (Default: “”)--recursive
: Whether to check files recursively in the given path (Default: false)
fs checksum
Usage: bin/alluxio fs checksum [path]
The checksum command outputs the md5 value of a file in Alluxio. This can be used to verify the contents of a file stored in Alluxio.
Examples:
# Compare the checksum values
# value from Alluxio filesystem
$ ./bin/alluxio fs checksum /LICENSE
md5sum: bf0513403ff54711966f39b058e059a3
# value from local filesystem
md5 LICENSE
MD5 (LICENSE) = bf0513403ff54711966f39b058e059a3
fs chgrp
Usage: bin/alluxio fs chgrp [group] [path] [flags]
The chgrp command changes the group of the file or directory in Alluxio. Alluxio supports file authorization with POSIX file permissions. The file owner or superuser can execute this command.
Flags:
--recursive
,-R
: change the group recursively for all files and directories under the given path (Default: false)
Examples:
# Change the group of a file
$ ./bin/alluxio fs chgrp alluxio-group-new /input/file1
fs chmod
Usage: bin/alluxio fs chmod [mode] [path] [flags]
The chmod command changes the permission of a file or directory in Alluxio. The permission mode is represented as an octal 3 digit value. Refer to https://en.wikipedia.org/wiki/Chmod#Numerical_permissions for a detailed description of the modes.
Flags:
--recursive
,-R
: change the permission recursively for all files and directories under the given path (Default: false)
Examples:
# Set mode 755 for /input/file
$ ./bin/alluxio fs chmod 755 /input/file1
fs chown
Usage: bin/alluxio fs chown <owner>[:<group>] <path> [flags]
The chown command changes the owner of a file or directory in Alluxio. The ownership of a file can only be altered by a superuser
Flags:
--recursive
,-R
: change the owner recursively for all files and directories under the given path (Default: false)
Examples:
# Change the owner of /input/file1 to alluxio-user
$ ./bin/alluxio fs chown alluxio-user /input/file1
fs compact
Usage: bin/alluxio fs compact [path] [flags]
The compact command compact the position write file.
Flags:
--commitTimeout
: Default is 1d. Uncommitted files (files with an mtime greater than the manifest’s mtime) are retained during compaction. You can configure this value to delete uncommitted files. For example, we can set it to 8h, meaning files with an mtime greater than 8 hours before the current time will be deleted, while files with an mtime less than 8 hours will be retained. (Default: “”)--deleteOrphanFrameFiles
: Default is false. Whether delete orphan frame files. (Default: false)--deleteOrphanFrameListFiles
: Default is false. Whether delete orphan frame list files. (Default: false)--deleteSnapshotsReferencedFiles
: Default is false. Whether delete the files referenced by the snapshots. If deleted, the files cannot be rolled back to a previous version. (Default: false)--exportPath
: Default is null. If the path is specified, during compaction, the frames will be merged into a complete file and output to the given path. (Default: “”)--fileLifecycle
: Default is 0. Unreferenced files will be deleted during compaction. You can configure this value to retain some recently created files. For example, we can set it to 8h, meaning files with an mtime (modification time) greater than 8 hours before the current time will be deleted, while files with an mtime less than 8 hours will be retained. (Default: “”)--maxRetainedSnapshots
: Default is 4. The maximum number of snapshots to be saved. When the number of snapshots exceeds this limit, the oldest snapshot will be deleted. (Default: “”)--parallelism
: Default is 1. The concurrency of compaction tasks. (Default: “”)--recursive
: Default is false. Whether compact given path recursively. (Default: false)
Examples:
# Compact file /log.txt
$ ./bin/alluxio fs compact [--deleteOrphanFrameListFiles] [--deleteOrphanFrameFiles] [--fileLifecycle 6h] [--commitTimeout 1h] [--maxRetainedSnapshots 4] [--deleteSnapshotsReferencedFiles] [--exportPath /log.compaction.txt] [--recursive] [--parallelism 8] /log.txt
fs cp
Usage: bin/alluxio fs cp [srcPath] [dstPath] [flags]
Copies a file or directory in the Alluxio filesystem or between local and Alluxio filesystems. The file:// scheme indicates a local filesystem path and the alluxio:// scheme or no scheme indicates an Alluxio filesystem path.
Flags:
--buffer-size
: Read buffer size when coping to or from local, with defaults of 64MB and 8MB respectively (Default: “”)--preserve
,-p
: Preserve file permission attributes when copying files; all ownership, permissions, and ACLs will be preserved (Default: false)--recursive
,-R
: True to copy the directory subtree to the destination directory (Default: false)--thread
: Number of threads used to copy files in parallel, defaults to 2 * CPU cores (Default: 0)
Examples:
# Copy within the Alluxio filesystem
$ ./bin/alluxio fs cp /file1 /file2
# Copy a local file to the Alluxio filesystem
$ ./bin/alluxio fs cp file:///file1 /file2
# Copy a file in Alluxio to local
$ ./bin/alluxio fs cp alluxio:///file1 file:///file2
# Recursively copy a directory within the Alluxio filesystem
$ ./bin/alluxio fs cp -R /dir1 /dir2
fs head
Usage: bin/alluxio fs head [path] [flags]
The head command prints the first 1KB of data of a file to the shell. Specifying the -c flag sets the number of bytes to print.
Flags:
--bytes
,-c
: Byte size to print (Default: “”)
Examples:
# Print first 2048 bytes of a file
$ ./bin/alluxio fs head -c 2048 /output/part-00000
fs location
Usage: bin/alluxio fs location [path]
Displays the list of hosts storing the specified file.
fs ls
Usage: bin/alluxio fs ls [path] [flags]
The ls command lists all the immediate children in a directory and displays their file info. Using ls on a file will only display the information for that specific file.
Flags:
--cache-filter
,-c
: Show the cacheability rules for metadata and data of the file based on the cache filter configuration; if cache filter is not enabled, show Disabled. Note the resolved cacheability are based on cache filter observed by the command line. (Default: false)--help
: help for this command (Default: false)--human-readable
,-h
: Print sizes in human readable format (Default: false)--list-dir-as-file
,-d
: List directories as files (Default: false)--recursive
,-R
: List subdirectories recursively (Default: false)
Examples:
# List immediate children of /s3/data
$ ./bin/alluxio fs ls /s3/data
fs mkdir
Usage: bin/alluxio fs mkdir [path1 path2 ...]
The mkdir command creates a new directory in the Alluxio filesystem. It is recursive and will create any parent directories that do not exist. Note that the created directory will not be created in the under storage system until a file in the directory is persisted to the underlying storage. Using mkdir on an invalid or existing path will fail.
Examples:
# Creating a folder structure
$ ./bin/alluxio fs mkdir /users
$ ./bin/alluxio fs mkdir /users/Alice
$ ./bin/alluxio fs mkdir /users/Bob
fs mv
Usage: bin/alluxio fs mv [srcPath] [dstPath]
The mv command moves a file or directory to another path in Alluxio. The destination path must not exist or be a directory. If it is a directory, the file or directory will be placed as a child of the directory. The command is purely a metadata operation and does not affect the data blocks of the file.
Examples:
# Moving a file
$ ./bin/alluxio fs mv /data/2014 /data/archives/2014
fs rm
Usage: bin/alluxio fs rm [path] [flags]
The rm command removes a file from Alluxio space and the under storage system. The file will be unavailable immediately after this command returns, but the actual data may be deleted a while later.
Flags:
--alluxio-only
: True to only remove data and metadata from Alluxio cache (Default: false)--recursive
,-R
: True to recursively remove files within the specified directory subtree (Default: false)--skip-ufs-check
,-U
: True to skip checking if corresponding UFS contents are in sync (Default: false)
Examples:
# Remove a file from Alluxio and the under storage system
$ ./bin/alluxio fs rm /tmp/unused-file
# Remove a file from Alluxio filesystem only
$ ./bin/alluxio fs rm --alluxio-only --skip-ufs-check /tmp/unused-file2
# Note it is recommended to use both --alluxio-only and --skip-ufs-check together in this situation
fs stat
Usage: bin/alluxio fs stat [path] [flags]
The stat command dumps the FileInfo representation of a file or a directory to the shell.
Flags:
--format
,-f
: Display info in the given format: “%N”: name of the file “%z”: size of file in bytes “%u”: owner “%g”: group name of owner “%i”: file id of the file “%y”: modification time in UTC in ‘yyyy-MM-dd HH:mm:ss’ format “%Y”: modification time as Unix timestamp in milliseconds “%b”: Number of blocks allocated for file (Default: “”)
Examples:
# Display file's stat
$ ./bin/alluxio fs stat /data/2015/logs-1.txt
# Display directory's stat
$ ./bin/alluxio fs stat /data/2015
# Display the size of file
$ ./bin/alluxio fs stat -f %z /data/2015/logs-1.txt
fs tail
Usage: bin/alluxio fs tail [path] [flags]
The tail command prints the last 1KB of data of a file to the shell. Specifying the –bytes flag sets the number of bytes to print.
Flags:
--bytes
: Byte size to print (Default: “”)
Examples:
# Print last 2048 bytes of a file
$ ./bin/alluxio fs tail --bytes 2048 /output/part-00000
fs test
Usage: bin/alluxio fs test [path] [flags]
Test a property of a path, returning 0 if the property is true, or 1 otherwise
Flags:
--dir
,-d
: Test if path is a directory (Default: false)--exists
,-e
: Test if path exists (Default: false)--file
,-f
: Test if path is a file (Default: false)--not-empty
,-s
: Test if path is not empty (Default: false)--zero
,-z
: Test if path is zero length (Default: false)
fs touch
Usage: bin/alluxio fs touch [path]
Create a 0 byte file at the specified path, which will also be created in the under file system
info
Retrieve and/or display info about the running Alluxio cluster
info cluster
Usage: bin/alluxio info cluster
Print the bound cluster information
info nodes
Usage: bin/alluxio info nodes
Show all registered workers’ status
info production
Usage: bin/alluxio info production
Print the production ID
info version
Usage: bin/alluxio info version
Print Alluxio version.
job
Command line tool for interacting with the job service.
job free
Usage: bin/alluxio job free [flags]
The free command triggers a scheduler job to free a directory and release cached pages in all workers.
Flags:
--batch-size
: [submit] The speed of free files per second per worker. If the value is 0 or empty, it will be set to the value of ‘alluxio.job.batch.size’ (Default: 0)--force
: [submit] Trigger free even if some workers are offline (Default: false)--format
: [progress] Format of output, either TEXT or JSON (Default: “”)--path
: (Required) [All] Source path of free operation--progress
: View progress of submitted job (Default: false)--recursive
: [submit] recursive free all files (Default: true)--stop
: Stop running job (Default: false)--submit
: Submit job (Default: false)--verbose
: [progress] Verbose output (Default: false)
Examples:
# Submit a free job to free path cache
$ ./bin/alluxio job free --path s3://bucket/path --submit
# View the progress of a submitted job
$ ./bin/alluxio job free --path s3://alluxio/path --progress
Progress for Free path file 's3://alluxio/path':
Job Id: b21ce9fb-f332-4d39-8bb4-554f9a4fa601
Job Submitted: Fri Feb 02 21:28:56 CST 2024
Job path: s3://alluxio/path
Job State: SUCCEEDED, finished at Fri Feb 02 21:29:01 CST 2024
Free Info : totalFile:4 totalByte:3072.00KB
Free Files Failed: 0
Free Bytes Failed: 0B
Free Files Succeeded: 4
Free Bytes Succeeded: 3072.00KB
# Stop a submitted job with best effort. Workers will stop freeing cache under the specified path as soon as possible.
$ ./bin/alluxio job free --path s3://alluxio/path --stop
job list
Usage: bin/alluxio job list [flags]
According to job type and job state list jobs.
Flags:
-
--job-state
: job state. RUNNINGVERIFYING STOPPED SUCCEEDED FAILED ALL (Default: “”) -
--job-type
: job type. LOADFREE COPY MOVE ALL (Default: “”)
Examples:
$./bin/alluxio job list --job-type all --job-state all
List job type:ALL state:ALL
{"mJobId":"1c723247-8381-4d69-bd29-b0f4fb34fc8d","mFreePath":"s3://alluxio/test-dir/","mJobType":"free","mJobState":"SUCCEEDED","mFreeBytes":81920000,"mFreeFileNums":20000,"mFreeFailFileNums":0,"mFreeFailBytes":0,"mFailInfos":{},"mTasks":{"WorkerInfo{id=0, identity=worker-1, address=WorkerNetAddress{host=hostname, containerHost=, rpcPort=29999, dataPort=29997, webPort=30000, domainSocketPath=, secureRpcPort=0, httpServerPort=0}, lastContactSec=0, state=LIVE, capacityBytes=0, usedBytes=0, startTimeMs=0, capacityBytesOnTiers=null, usedBytesOnTiers=null, version=, revision=}":"SUCCESS"},"mVerbose":true,"mStartTime":1722490595894,"mEndTime":1722490606649}
{"mPath":"s3://alluxio/test-dir/","mIndexFile":"","mJobType":"load","mVerbose":true,"mJobState":"SUCCEEDED","mVerificationEnabled":false,"mSkippedByteCount":0,"mLoadedByteCount":81920000,"mScannedInodesCount":20000,"mLoadedNonEmptyFileCopiesCount":20000,"mTotalByteCount":81920000,"mThroughput":1462857,"mFailureFilesPercentage":0.0,"mFailureSubTasksPercentage":0.0,"mRetrySubTasksPercentage":0.0,"mFailedFileCount":0,"mRecentFailedSubtasksWithReasons":[],"mRecentRetryingSubtasksWithReasons":[],"mSkipIfExists":false,"mMetadataOnly":false,"mRunningStage":"","mRetryDeadLetterQueueSize":0,"mTimeElapsed":52693,"mSegmentEnabled":false}
job load
Usage: bin/alluxio job load [flags]
The load command moves data from the under storage system into Alluxio storage. For example, load can be used to prefetch data for analytics jobs. If load is run on a directory, files in the directory will be recursively loaded.
Flags:
--bandwidth
: [submit] Single worker read bandwidth limit (Default: “”)--batch-size
: [submit] # of batch size to load at every worker. If the value is 0 or empty, it will be set to the value of alluxio.job.batch.size (Default: 0)--file-filter-regx
: [submit] Skip files that match the regx pattern (Default: “”)--force
: [submit] Trigger metadata sync even if some workers are offline (Default: false)--format
: [progress] Format of output, either TEXT or JSON (Default: “”)--index-file
: [all] Source path of the index file for load operation (Default: “”)--metadata-only
: [submit] Only load file metadata (Default: false)--partial-listing
: [submit] Use partial directory listing, initializing load before reading the entire directory but cannot report on certain progress details (Default: false)--path
: [all] Source path of load operation (Default: “”)--progress
: View progress of submitted job (Default: false)--replicas
: [submit] # of replicas to load (Default: 1)--skip-if-exists
: [submit] Skip existing fullly cached files (Default: false)--skip-quota-check
: [submit] skip quota check and force submit load hob (Default: false)--stop
: Stop running job (Default: false)--submit
: Submit job (Default: false)--verbose
: [progress] Verbose output (Default: false)--verify
: [submit] Run verification when load finishes and load new files if any (Default: false)
Examples:
# Submit a load job
$ ./bin/alluxio job load --path <full-ufs-path> --submit
OR
$ ./bin/alluxio job load --index-file <full-ufs-path> --submit
If --index-file is used, load command will read the index file and load the file paths in it.
Both file and directory are supported. In the index file, each line represents an UFS path.
An example index file looks like the following:
hdfs://cluster/root/file
s3://bucket/directory/
#file:///ignored
The hdfs path will be loaded as a single file,
the s3 path will be loaded recursively as a directory,the s3 path will be loaded recursively as a directory,
because it ends with the separator / ,
while the local file will be ignored as it starts with # and considered as a line of comment.
# View the progress of a submitted job
$ ./bin/alluxio job load --path <full-ufs-path> --progress
OR
$ ./bin/alluxio job load --index-file <full-ufs-path> --progress
# Example output
Progress for loading path '/path':
Settings: bandwidth: unlimited verify: false
Job State: SUCCEEDED
Files Processed: 1000
Bytes Loaded: 125.00MB
Throughput: 2509.80KB/s
Block load failure rate: 0.00%
Files Failed: 0
# Stop a submitted job
$ ./bin/alluxio job load --path <full-ufs-path> --stop
OR
$ ./bin/alluxio job load --index-file <full-ufs-path> --stop
journal
Journal related operations
journal checkpoint
Usage: bin/alluxio journal checkpoint
The checkpoint command creates a checkpoint the leading Alluxio master’s journal. This command is mainly used for debugging and to avoid master journal logs from growing unbounded. Checkpointing requires a pause in master metadata changes, so use this command sparingly to avoid interfering with other users of the system.
journal format
Usage: bin/alluxio journal format
The format command formats the local Alluxio master’s journal.
Warning: Formatting should only be called while the cluster is not running.
journal read
Usage: bin/alluxio journal read [flags]
The read command parses the current journal and outputs a human readable version to the local folder. This command may take a while depending on the size of the journal.
Note: This command requies that the Alluxio cluster is NOT running.
Flags:
--end
: end log sequence number (exclusive) (Default: -1)--input-dir
: input directory on-disk to read the journal content from (Default: “”)--master
: name of the master class (Default: “”)--output-dir
: output directory to write journal content to (Default: “”)--start
: start log sequence number (inclusive) (Default: 0)
Examples:
$ ./bin/alluxio journal read
# output
Dumping journal of type EMBEDDED to /Users/alluxio/journal_dump-1602698211916
2020-10-14 10:56:51,960 INFO RaftStorageDirectory - Lock on /Users/alluxio/alluxio/journal/raft/02511d47-d67c-49a3-9011-abb3109a44c1/in_use.lock acquired by nodename 78602@alluxio-user
2020-10-14 10:56:52,254 INFO RaftJournalDumper - Read 223 entries from log /Users/alluxio/alluxio/journal/raft/02511d47-d67c-49a3-9011-abb3109a44c1/current/log_0-222.
license
Check and manage license status
license check-expiration
Usage: bin/alluxio license check-expiration
Validates the license and print warnings if any constraint is about to be exceeded
license show
Usage: bin/alluxio license show [flags]
Show the details of the license based on the prod jar and configuration
Flags:
--output
,-o
: Output format, could be json/yaml (Default: “”)
license status
Usage: bin/alluxio license status [flags]
List the current status of the cluster that license may use
Flags:
--raw
: Output raw JSON data instead of human-readable format for bytes, datetime, and duration. (Default: false)
license update
Usage: bin/alluxio license update [flags]
Use the license from the current site properties file to update
Flags:
--process
: The process you want to update with the new license (Default: “”)
mount
The mount command manges the mapping from under storage path to an Alluxio path, where files and folders created in Alluxio space under the path will be backed by a corresponding file or folder in the under storage path.
mount add
Usage: bin/alluxio mount add [flags]
The add command can be used to make data in another storage system available in Alluxio.
Note that the –readonly flag mounts are useful to prevent accidental write operations. If multiple Alluxio satellite clusters mount a remote storage cluster which serves as the central source of truth, the –readonly option could help prevent any write operations on the satellite cluster from wiping out the remote storage.
To connect to the UFS for a mount point, Alluxio looks for the corresponding connector under ${ALLUXIO_HOME}/lib/ and will use the first one that supports the path. The connector jars look like lib/alluxio-underfs-hdfs-2.7.1.jar. The logic to decide whether a connector supports a path depends on the UnderFileSystemFactory implementation. When there are multiple connectors for the same UFS, like lib/alluxio-underfs-hdfs-2.7.1.jar, lib/alluxio-underfs-hdfs-2.7.1-patch1.jar, lib/alluxio-underfs-hdfs-2.7.1-patch2.jar, the option “alluxio.underfs.strict.version.match.enabled” can be used to make sure the correct one is picked up. For example, if the HDFS is running with 2.7.1-patch1, you can use “alluxio.underfs.version” and “alluxio.underfs.strict.version.match.enabled=true” to ensure “lib/alluxio-underfs-hdfs-2.7.1-patch1.jar” is used to connect to the target HDFS at hdfs://ns1/
Flags:
--master
: Set true to create mount point tracked by Alluxio master (Default: false)--option
: Configuration options, in the form of= , associated with the mount point, such as credentials (Default: []) --path
: (Required) Alluxio path to mount onto--shared
: Sets the permission bits of the mount point to be accessible for all Alluxio users (Default: false)--ufs-uri
: (Required) UFS URI to mount
Examples:
# Add a mount point at /hdfs to access hdfs://host1:9000/data/
# Note that the mount point must be under the root directory
$ ./bin/alluxio mount add --path /hdfs --ufs-uri hdfs://host1:9000/data/
# Add a read only mount that is accessible to all Alluxio users
$ ./bin/alluxio mount add --path /hdfs2 --ufs-uri hdfs://host2:9000/data/ --shared --readonly
# Add a S3 bucket mount with S3 credentials required to access the bucket
$ ./bin/alluxio mount add --path /s3 --ufs-uri s3://data-bucket/ --option s3a.accessKeyId=<accessKey> --option s3a.secretKey=<secretKey> --option alluxio.underfs.s3.region=us-east-1
# Add a HDFS mount using a specific UFS jar in lib/
$ ./bin/alluxio mount add --path /ns1 --ufs-uri hdfs://ns1/ --option alluxio.underfs.version=2.7.1-patch1 --option alluxio.underfs.strict.version.match.enabled=true
mount list
Usage: bin/alluxio mount list [flags]
List all known mount points set on the Alluxio filesystem
Flags:
--master
: Set true to list mount points under the master based registration (Default: false)
Examples:
# List mount points
$ ./bin/alluxio mount list
mount remove
Usage: bin/alluxio mount remove [flags]
Removes the mount point at the specified path
Flags:
--master
: Set true to delete mount point under the master based registration (Default: false)--path
: (Required) Alluxio path to unmount
Examples:
# Remove mount point at /mnt/hdfs
$ ./bin/alluxio mount remove --path /mnt/hdfs
process
Start/stop cluster processes or remove workers
process remove-worker
Usage: bin/alluxio process remove-worker [flags]
Remove given worker from the cluster, so that clients and other workers will not consider the removed worker for services. The worker must have been stopped before it can be safely removed from the cluster.
Flags:
--name
,-n
: (Required) Worker id
process start
Usage: bin/alluxio process start [flags]
Starts a single process locally or a group of similar processes across the cluster. For starting a group, it is assumed the local host has passwordless SSH access to other nodes in the cluster. The command will parse the hostnames to run on by reading the conf/coordinator and conf/workers files, depending on the process type.
Flags:
--async
,-a
: Asynchronously start processes without monitoring for start completion (Default: false)--console-log
,-c
: Log output to stdout in addition to log file (Default: false)--direct
,-d
: (For use in docker) Directly run the start command, skipping all other steps and avoid using nohup to launch (Default: false)--skip-kill-prev
,-N
: Avoid killing previous running processes when starting (Default: false)
process stop
Usage: bin/alluxio process stop [flags]
Stops a single process locally or a group of similar processes across the cluster. For stopping a group, it is assumed the local host has passwordless SSH access to other nodes in the cluster. The command will parse the hostnames to run on by reading the conf/coordinator and conf/workers files, depending on the process type.
Flags:
--soft
,-s
: Soft kill only, don’t forcibly kill the process (Default: false)
quota
The quota command manges all quota definitions in Alluxio system. Alluxio supports setting quota on directories in Alluxio path.
quota add
Usage: bin/alluxio quota add [flags]
Use the add command to set a new quota definition on a directory in Alluxio path. The quota definition will be applied to the total cache in all Alluxio workers.
Flags:
--directory
: (Required) Alluxio path to set quota on--quota-size
: (Required) Size of the quota definition, eg. 10GB, 1TB
Examples:
# Add a quota of 10GB to the directory /s3/ in Alluxio namespace
# The cache in all Alluxio workers combined will be limited to 10GB
$ bin/alluxio quota add --directory /s3/ --quota-size 10GB
quota list
Usage: bin/alluxio quota list [flags]
Use the list command to list the current usage of all quotas defined in Alluxio. You can also use the –interval option to refresh the quota usage information every specified interval.
Flags:
--interval
: Interval to refresh the quota usage information. (Default: “”)
Examples:
# Lists current quota usage
$ bin/alluxio quota list
Alluxio path: /local, Quota capacity bytes: 11534336, Used bytes: 10486645, State: Available
Alluxio path: /s3/data, Quota capacity bytes: 10737418240, Used bytes: Calculating, State: Available
# Keeps running until interrupted by user by pressing ctrl+C
$ bin/alluxio quota list --interval 5s
Polling quota usage summary from the master every 5s
Alluxio path: /local, Quota capacity bytes: 11534336, Used bytes: 10486645, State: Available
Alluxio path: /s3/data, Quota capacity bytes: 10737418240, Used bytes: Calculating, State: Available
Alluxio path: /local, Quota capacity bytes: 11534336, Used bytes: 10486645, State: Available
Alluxio path: /s3/data, Quota capacity bytes: 10737418240, Used bytes: Calculating, State: Available
Alluxio path: /local, Quota capacity bytes: 11534336, Used bytes: 10486645, State: Available
Alluxio path: /s3/data, Quota capacity bytes: 10737418240, Used bytes: 10486645, State: Available
quota remove
Usage: bin/alluxio quota remove [flags]
Use the remove command to remove an existing quota definition on an Alluxio path.
Flags:
--directory
: (Required) Alluxio directory to remove quota on
Examples:
$ bin/alluxio quota remove --directory /s3/
Successfully removed quota definition for path /s3/data.
quota update
Usage: bin/alluxio quota update [flags]
Use the update command to update an existing quota definition.
Flags:
--directory
: (Required) Alluxio path to set quota on--force
: Whether to force the update of the quota, even if it violates the existing space usage (Default: false)--quota-size
: (Required) Size of the quota definition, eg. 10GB, 1TB
Examples:
$ bin/alluxio quota update --directory /local/data/ --quota-size 100GB
security
Security server related commands
security token
Usage: bin/alluxio security token
Get one token from the security service