alluxio.job.migrate

Class MigrateDefinition

java.lang.Object

alluxio.job.AbstractVoidJobDefinition<MigrateConfig,ArrayList<MigrateCommand>>

alluxio.job.migrate.MigrateDefinition

All Implemented Interfaces:

JobDefinition<MigrateConfig,ArrayList<MigrateCommand>,SerializableVoid>

public final class MigrateDefinition
extends AbstractVoidJobDefinition<MigrateConfig,ArrayList<MigrateCommand>>

A job that migrates a source path to a destination path. The source and the destination can be across mount points. If the destination exists, the source and destination must either both be files or both be directories, and the overwrite configuration option must be set. If the destination does not exist, its parent must be a directory and the destination will be created by the migrate command. If migrating a directory to an existing directory causes files to conflict, the migrated files will replace the existing files. Unlike Unix mv or cp, the source will not be nested inside the destination when the destination is a directory. This makes it so that the migrate job is idempotent when the overwrite flag is set. Suppose we have this directory structure, where a to e are directories and f1 to f3 are files: ├── a │ ├── e │ │ └── f2 │ └── f1 └── b └── d └── e └── f3 Migrating a to b with source deleted (mv) will result in ├── b ├── d ├── e │ └── f2 │ └── f3 └── f1 Migrating a to b with source kept (cp) will result in ├── a │ ├── e │ │ └── f2 │ └── f1 └── b ├── d ├── e │ └── f2 │ └── f3 └── f1

Constructor Summary

Constructors

Constructor and Description
MigrateDefinition() Constructs a new MigrateDefinition.

Method Summary

Modifier and Type	Method and Description
Class<MigrateConfig>	getJobConfigClass()
SerializableVoid	runTask(MigrateConfig config, ArrayList<MigrateCommand> commands, RunTaskContext context) Runs the task in the executor.
Map<WorkerInfo,ArrayList<MigrateCommand>>	selectExecutors(MigrateConfig config, List<WorkerInfo> jobWorkerInfoList, SelectExecutorsContext context) Selects the workers to run the task.

Methods inherited from class alluxio.job.AbstractVoidJobDefinition

join

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

MigrateDefinition

public MigrateDefinition()

Constructs a new MigrateDefinition.

Method Detail

selectExecutors

public Map<WorkerInfo,ArrayList<MigrateCommand>> selectExecutors(MigrateConfig config,
                                                                 List<WorkerInfo> jobWorkerInfoList,
                                                                 SelectExecutorsContext context)
                                                          throws Exception

Selects the workers to run the task. Assigns each worker to migrate whichever files it has the most blocks for. If no worker has blocks for a file, a random worker is chosen.

Parameters:

config - the job configuration

jobWorkerInfoList - the list of available workers' information

context - the context containing information used to select executors

Returns:

a map of selected workers to the parameters to pass along

Throws:

Exception - if any error occurs

runTask

public SerializableVoid runTask(MigrateConfig config,
                                ArrayList<MigrateCommand> commands,
                                RunTaskContext context)
                         throws Exception

Runs the task in the executor. Migrates the file specified in the config to the configured path. If the destination path is a directory, the file is migrated inside that directory.

Parameters:

config - the job configuration

commands - the arguments passed in

context - the context containing information used to execute a task

Returns:

the task result

Throws:

Exception - if any error occurs

getJobConfigClass

public Class<MigrateConfig> getJobConfigClass()

Returns:

the class of the associated JobConfig