Admin Command Line Interface

Slack Docker Pulls GitHub edit source

Alluxio’s admin command line interface provides admins with operations to manage the Alluxio filesystem. You can invoke the following command line utility to get all the subcommands:

$ ./bin/alluxio fsadmin
Usage: alluxio fsadmin [generic options]
       [backup]
       [checkpoint]
       [doctor [category]]
       [report]
       [ufs --mode <noAccess/readOnly/readWrite> <ufsPath>]
       ...

Operations

backup

The backup command backs up all Alluxio metadata to the backup directory configured on the leader master.

Back up to the default backup folder /alluxio_backups of the root under storage system. This default backup directory can be configured by setting alluxio.master.backup.directory.

./bin/alluxio fsadmin backup
Successfully backed up journal to hdfs://host:port/alluxio_backups/alluxio-backup-2018-5-29-1527644810.gz

Note that the user running the backup command need to have write permission to the backup folder of root under storage system.

Back up to a specific directory in the root under storage system.

./bin/alluxio fsadmin backup /alluxio/special_backups
Successfully backed up journal to hdfs://host:port/alluxio/special_backups/alluxio-backup-2018-5-29-1527644810.gz

Back up to a specific directory on the leading master’s local filesystem.

./bin/alluxio fsadmin backup /opt/alluxio/backups/ --local
Successfully backed up journal to /opt/alluxio/backups/alluxio-backup-2018-5-29-1527644810.gz on master Master2

journal

The journal command provides several sub-commands for journal management.

quorum: is used to query and manage embedded journal powered leader election.

# Get information on existing state of the `MASTER` or `JOB_MASTER` leader election quorum.
$ ./bin/alluxio fsadmin journal quorum info -domain <MASTER | JOB_MASTER>
# Remove a member from leader election quorum.
$ ./bin/alluxio fsadmin journal quorum remove -domain <MASTER | JOB_MASTER> -address <Member_Address>

checkpoint: is used to create a checkpoint in the primary master journal system.

This command is mainly used for debugging and to avoid master journal logs from growing unbounded.

Checkpointing requires a pause in master metadata changes, so use this command sparingly to avoid interfering with other users of the system.

$ ./bin/alluxio fsadmin journal checkpoint

doctor

The doctor command gives recommendations and warnings. It can diagnose inconsistent configurations across different Alluxio nodes as well as alert the operator when worker storage volumes are missing.

# shows server-side configuration errors and warnings
$ ./bin/alluxio fsadmin doctor configuration
# shows worker storage health errors and warnings
$ ./bin/alluxio fsadmin doctor storage

getBlockInfo

The getBlockInfo command provides the block information and file path of a block id. It is primarily intended to assist power users in debugging their system.

$ ./bin/alluxio fsadmin getBlockInfo <block_id>
BlockInfo{id=16793993216, length=6, locations=[BlockLocation{workerId=8265394007253444396, address=WorkerNetAddress{host=local-mbp, rpcPort=29999, dataPort=29999, webPort=30000, domainSocketPath=, tieredIdentity=TieredIdentity(node=local-mbp, rack=null)}, tierAlias=MEM, mediumType=MEM}]}
This block belongs to file {id=16810770431, path=/test2}

metrics

The metrics command provides operations for Alluxio metrics system.

metrics clear will clear the metrics stored in the whole alluxio cluster. This command is useful when getting metrics information in short-term testing. It should be used sparingly as it may affect the current metrics recording and reporting which may lead to metrics incorrectness and affect worker/client heartbeats with leading master.

If --master option is used, all the metrics stored in Alluxio leading master will be cleared. If --workers <WORKER_HOSTNAME_1>,<WORKER_HOSTNAME_2> is used, metrics in specific workers will be cleared.

If you are clearing metrics of a large Alluxio cluster with many workers, you can use the --parallelism <#> option to submit # of worker metrics clearance job in parallel. For example, if your cluster has 200 workers, persisting with a parallelism factor of 10 will clear metrics of 10 workers at a time until all metrics in 200 workers are cleared.

# Clear metrics of the whole alluxio cluster including leading master and workers
$ ./bin/alluxio fsadmin metrics clear
# Clear metrics of alluxio leading master
$ ./bin/alluxio fsadmin metrics clear --master
# Clear metrics of specific workers
$ ./bin/alluxio fsadmin metrics clear --workers <WORKER_HOSTNAME_1>,<WORKER_HOSTNAME_2>
# Clear metrics of an alluxio cluster with many workers in parallel
$ ./bin/alluxio fsadmin metrics clear --parallelism 10

report

The report command provides Alluxio running cluster information.

If no argument is passed in, report will report the leading master, worker number, and capacity information.

$ ./bin/alluxio fsadmin report
Alluxio cluster summary:
    Master Address: localhost:19998
    Zookeeper Enabled: false
    Live Workers: 1
    Lost Workers: 0
    Total Capacity: 10.45GB
    Used Capacity: 0B
    (only a subset of the results is shown)

report capacity will report Alluxio cluster capacity information for different subsets of workers:

  • -live Live workers
  • -lost Lost workers
  • -workers <worker_names> Specified workers, host names or ip addresses separated by ,.
# Capacity information of all workers
$ ./bin/alluxio fsadmin report capacity
# Capacity information of live workers
$ ./bin/alluxio fsadmin report capacity -live
# Capacity information of specified workers
$ ./bin/alluxio fsadmin report capacity -workers AlluxioWorker1,127.0.0.1

report metrics will report the metrics stored in the leading master which includes leading master process metrics and aggregated cluster metrics.

$ ./bin/alluxio fsadmin report metrics

report ufs will report all the mounted under storage system information of Alluxio cluster.

$ ./bin/alluxio fsadmin report ufs
Alluxio under storage system information:
hdfs://localhost:9000/ on / (hdfs, capacity=-1B, used=-1B, not read-only, not shared, properties={})

report jobservice will report a summary of the job service.

$ ./bin/alluxio fsadmin report jobservice
Status: CREATED   Count: 0
Status: CANCELED  Count: 0
Status: FAILED    Count: 1
Status: RUNNING   Count: 118
Status: COMPLETED Count: 223

10 Most Recently Modified Jobs:
Timestamp: 10-24-2019 17:15:25:014       Id: 1571936656844       Name: Persist             Status: COMPLETED
Timestamp: 10-24-2019 17:15:24:340       Id: 1571936656957       Name: Persist             Status: RUNNING
(only a subset of the results is shown)

10 Most Recently Failed Jobs:
Timestamp: 10-24-2019 17:15:22:946       Id: 1571936656839       Name: Persist             Status: FAILED

ufs

The ufs command provides options to update attributes of a mounted under storage. The option mode can be used to put an under storage in maintenance mode. Certain operations can be restricted at this moment.

For example, an under storage can enter readOnly mode to disallow write operations. Alluxio will not attempt any write operations on the under storage.

$ ./bin/alluxio fsadmin ufs --mode readOnly hdfs://ns

The fsadmin ufs subcommand takes a UFS URI as an argument. The argument should be a root UFS URI like hdfs://<name-service>/, and not hdfs://<name-service>/<folder>.

pathConf

The pathConf command manages path defaults.

list

pathConf list lists paths configured with path defaults.

$ ./bin/alluxio fsadmin pathConf list

/a
/b

The above command shows that there are path defaults set for paths with prefix /a and /b.

show

pathConf show shows path defaults for a specific path.

It has two modes:

  1. without option --all, only show path defaults set for the specific path;
  2. with option --all, show path defaults set for all paths that are prefixes of the specified path.

For example, suppose path defaults property1=value1 is set for /a, and property2=value2 is set for /a/b.

Then without --all, only properties for /a/b are shown:

$ ./bin/alluxio fsadmin pathConf show /a/b

property2=value2

With --all, since /a is a prefix of /a/b, properties for both /a and /a/b are shown:

$ ./bin/alluxio fsadmin pathConf show --all /a/b

property1=value1
property2=value2

add

pathConf add adds or updates path defaults, only properties with scope equal to or broader than the client scope can be set as path defaults.

$ ./bin/alluxio fsadmin pathConf add --property property1=value1 --property property2=value2 /tmp

The above command adds two properties as path defaults for paths with prefix /tmp.

$ ./bin/alluxio fsadmin pathConf add --property property1=value2 /tmp

The above command updates the value of property1 from value1 to value2 for path defaults of /tmp.

remove

pathConf remove removes properties from path defaults for a path.

$ ./bin/alluxio fsadmin pathConf remove --keys property1,property2 /tmp

The above command removes properties with key property1 and property2 from path defaults for paths with prefix /tmp.