alluxio.master.job

Class CopyJob

java.lang.Object

alluxio.master.job.AbstractJob<CopyJob.CopyTask>

alluxio.master.job.CopyJob

All Implemented Interfaces:

Job<CopyJob.CopyTask>

@NotThreadSafe
public class CopyJob
extends AbstractJob<CopyJob.CopyTask>

Copy job that copy a file or a directory from source to destination. This class should only be manipulated from the scheduler thread in Scheduler thus the state changing functions are not thread safe. TODO() as task within this class is running on multithreaded context, make thread unsafe places to be thread safe in future.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	CopyJob.CopyTask Loads blocks in a UFS through an Alluxio worker.

Field Summary

Fields

Modifier and Type	Field and Description
static com.codahale.metrics.Counter	COPY_FAIL_FILE_COUNT
static com.codahale.metrics.Counter	COPY_FILE_COUNT
static com.codahale.metrics.Meter	COPY_RATE
static com.codahale.metrics.Counter	COPY_SIZE
static com.codahale.metrics.Counter	COPY_SKIP_FILE_COUNT
static com.codahale.metrics.Counter	JOB_COPY_FAIL
static com.codahale.metrics.Counter	JOB_COPY_SUCCESS
static java.util.function.Predicate<FileInfo>	QUALIFIED_FILE_FILTER
static String	TYPE

Fields inherited from class alluxio.master.job.AbstractJob

mEndTime, mJobId, mRetryTaskList, mStartTime, mState, mTaskIdGenerator, mUser, mWorkerAssignPolicy

Constructor Summary

Constructors

Constructor and Description
CopyJob(String src, String dst, boolean overwrite, Optional<String> user, String jobId, OptionalLong bandwidth, boolean usePartialListing, boolean verificationEnabled, boolean checkContent, Iterable<FileInfo> fileIterable, Optional<Job.FileFilter> filter) Constructor.

Method Summary

Modifier and Type	Method and Description
void	addCopiedBytes(long bytes) Add bytes to total loaded bytes.
void	addFailure(String src, String message, int code) Add a block to failure summary.
void	addSkip() Add a skip to metrics.
boolean	addToRetry(Route route) Add a route to retry later.
boolean	equals(Object o)
void	failJob(AlluxioRuntimeException reason) Set load state to FAILED with given reason.
OptionalLong	getBandwidth() Get bandwidth.
JobDescription	getDescription()
long	getDurationInSec() Get duration in seconds.
List<Route>	getNextRoutes(int count) Get next batch of blocks.
List<CopyJob.CopyTask>	getNextTasks(Collection<WorkerInfo> workers) get next load task.
String	getProgress(JobProgressReportFormat format, boolean verbose) Get job progress.
String	getSrc()
boolean	hasFailure()
int	hashCode()
void	initiateVerification() Initiate a verification pass.
boolean	isCurrentPassDone() Check whether the current pass is finished.
boolean	isHealthy() Check whether the job is healthy.
boolean	isVerificationEnabled() Is verification enabled.
boolean	needVerification() Is verification enabled.
boolean	processResponse(CopyJob.CopyTask task) process task result.
void	setJobSuccess() set job as success.
void	setVerificationEnabled(boolean enableVerification) Enable verification.
Journal.JournalEntry	toJournalEntry()
String	toString()
void	updateBandwidth(OptionalLong bandwidth) Update bandwidth.

Methods inherited from class alluxio.master.job.AbstractJob

getEndTime, getJobId, getJobState, getWorkerAssignPolicy, initializeJob, isDone, isRunning, onTaskSubmitFailure, setEndTime, setJobState, setStartTime, setWorkerAssignPolicy

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

TYPE

public static final String TYPE

See Also:

Constant Field Values

QUALIFIED_FILE_FILTER

public static final java.util.function.Predicate<FileInfo> QUALIFIED_FILE_FILTER

JOB_COPY_SUCCESS

public static final com.codahale.metrics.Counter JOB_COPY_SUCCESS

JOB_COPY_FAIL

public static final com.codahale.metrics.Counter JOB_COPY_FAIL

COPY_FILE_COUNT

public static final com.codahale.metrics.Counter COPY_FILE_COUNT

COPY_FAIL_FILE_COUNT

public static final com.codahale.metrics.Counter COPY_FAIL_FILE_COUNT

COPY_SKIP_FILE_COUNT

public static final com.codahale.metrics.Counter COPY_SKIP_FILE_COUNT

COPY_SIZE

public static final com.codahale.metrics.Counter COPY_SIZE

COPY_RATE

public static final com.codahale.metrics.Meter COPY_RATE

Constructor Detail

CopyJob

public CopyJob(String src,
               String dst,
               boolean overwrite,
               Optional<String> user,
               String jobId,
               OptionalLong bandwidth,
               boolean usePartialListing,
               boolean verificationEnabled,
               boolean checkContent,
               Iterable<FileInfo> fileIterable,
               Optional<Job.FileFilter> filter)

Constructor.

Parameters:

src - file source

dst - file destination

overwrite - whether to overwrite the file

user - user for authentication

jobId - job identifier

bandwidth - bandwidth

usePartialListing - whether to use partial listing

verificationEnabled - whether to verify the job after loaded

checkContent - whether to check content

fileIterable - file iterable

filter - file filter

Method Detail

getSrc

public String getSrc()

Returns:

source file path

getDescription

public JobDescription getDescription()

Returns:

the description of the copy job. The path is the "src:dst" format

getBandwidth

public OptionalLong getBandwidth()

Get bandwidth.

Returns:

the allocated bandwidth

updateBandwidth

public void updateBandwidth(OptionalLong bandwidth)

Update bandwidth.

Parameters:

bandwidth - new bandwidth

isVerificationEnabled

public boolean isVerificationEnabled()

Is verification enabled.

Returns:

whether verification is enabled

needVerification

public boolean needVerification()

Is verification enabled.

Returns:

whether verification is enabled

setVerificationEnabled

public void setVerificationEnabled(boolean enableVerification)

Enable verification.

Parameters:

enableVerification - whether to enable verification

failJob

public void failJob(AlluxioRuntimeException reason)

Set load state to FAILED with given reason.

Parameters:

reason - failure exception

setJobSuccess

public void setJobSuccess()

Description copied from interface: Job

set job as success.

addCopiedBytes

public void addCopiedBytes(long bytes)

Add bytes to total loaded bytes.

Parameters:

bytes - bytes to be added to total

getProgress

public String getProgress(JobProgressReportFormat format,
                          boolean verbose)

Description copied from interface: Job

Get job progress.

Parameters:

format - progress report format

verbose - whether to include detailed information

Returns:

job progress report

equals

public boolean equals(Object o)

Overrides:

equals in class Object

hashCode

public int hashCode()

Overrides:

hashCode in class Object

isHealthy

public boolean isHealthy()

Description copied from interface: Job

Check whether the job is healthy.

Returns:

true if the job is healthy, false if not

isCurrentPassDone

public boolean isCurrentPassDone()

Description copied from interface: Job

Check whether the current pass is finished.

Returns:

true if the current pass of job is finished, false if not

initiateVerification

public void initiateVerification()

Description copied from interface: Job

Initiate a verification pass. This will re-list the directory and find any unfinished files / tasks and try to execute them again.

getNextTasks

public List<CopyJob.CopyTask> getNextTasks(Collection<WorkerInfo> workers)

get next load task.

Parameters:

workers - workerInfos

Returns:

the next task to run. If there is no task to run, return empty

getNextRoutes

public List<Route> getNextRoutes(int count)

Get next batch of blocks.

Parameters:

count - number of blocks

Returns:

list of blocks

addToRetry

public boolean addToRetry(Route route)

Add a route to retry later.

Parameters:

route - the route to retry

Returns:

whether the block is successfully added

addFailure

public void addFailure(String src,
                       String message,
                       int code)

Add a block to failure summary.

Parameters:

src - the source path of the file that failed

message - failure message

code - status code for exception

addSkip

public void addSkip()

Add a skip to metrics.

toString

public String toString()

Overrides:

toString in class Object

toJournalEntry

public Journal.JournalEntry toJournalEntry()

Returns:

job journal entry

getDurationInSec

public long getDurationInSec()

Get duration in seconds.

Returns:

job duration in seconds

processResponse

public boolean processResponse(CopyJob.CopyTask task)

Description copied from interface: Job

process task result.

Parameters:

task - task containing result future

Returns:

success if successfully process task result, otherwise return failure

hasFailure

public boolean hasFailure()

Returns:

whether the job has failed tasks