alluxio.master.job

Class LoadJob

java.lang.Object

alluxio.master.job.AbstractJob<LoadJob.LoadTask>

alluxio.master.job.LoadJob

All Implemented Interfaces:

Job<LoadJob.LoadTask>

@NotThreadSafe
public class LoadJob
extends AbstractJob<LoadJob.LoadTask>

Load job that loads a file or a directory into Alluxio. This class should only be manipulated from the scheduler thread in Scheduler thus the state changing functions are not thread safe. Deprecated class, refer to DoraLoadJob

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	LoadJob.LoadTask Loads blocks in a UFS through an Alluxio worker.

Field Summary

Fields

Modifier and Type	Field and Description
static com.codahale.metrics.Counter	JOB_LOAD_BLOCK_COUNT
static com.codahale.metrics.Counter	JOB_LOAD_BLOCK_FAIL
static com.codahale.metrics.Counter	JOB_LOAD_BLOCK_SIZE
static com.codahale.metrics.Counter	JOB_LOAD_FAIL
static com.codahale.metrics.Meter	JOB_LOAD_RATE
static com.codahale.metrics.Counter	JOB_LOAD_SUCCESS
static java.util.function.Predicate<FileInfo>	QUALIFIED_FILE_FILTER
static String	TYPE

Fields inherited from class alluxio.master.job.AbstractJob

mEndTime, mJobId, mRetryTaskList, mStartTime, mState, mTaskIdGenerator, mUser, mWorkerAssignPolicy

Constructor Summary

Constructors

Constructor and Description
LoadJob(String path, Optional<String> user, String jobId, OptionalLong bandwidth, boolean usePartialListing, boolean verificationEnabled, Iterable<FileInfo> fileIterable) Constructor.
LoadJob(String path, String user, OptionalLong bandwidth, FileIterable fileIterator) Constructor.

Method Summary

Modifier and Type	Method and Description
void	addBlockFailure(Block block, String message, int code) Add a block to failure summary.
boolean	addBlockToRetry(Block block) Add a block to retry later.
void	addLoadedBytes(long bytes) Add bytes to total loaded bytes.
boolean	equals(Object o)
void	failJob(AlluxioRuntimeException reason) Set load state to FAILED with given reason.
OptionalLong	getBandwidth() Get bandwidth.
long	getCurrentBlockCount() Get the processed block count in the current loading pass.
JobDescription	getDescription()
long	getDurationInSec() Get duration in seconds.
List<Block>	getNextBatchBlocks(int count) Get next batch of blocks.
List<LoadJob.LoadTask>	getNextTasks(Collection<WorkerInfo> workers) get next load task.
String	getPath() Get load file path.
String	getProgress(JobProgressReportFormat format, boolean verbose) Get job progress.
boolean	hasFailure()
int	hashCode()
void	initiateVerification() Initiate a verification pass.
boolean	isCurrentPassDone() Check whether the current pass is finished.
boolean	isHealthy() Check whether the job is healthy.
boolean	isVerificationEnabled() Is verification enabled.
boolean	needVerification() Is verification enabled.
void	onTaskSubmitFailure(Task<?> task) Define how to process task that gets rejected when scheduler tried to kick off.
boolean	processResponse(LoadJob.LoadTask loadTask) process task result.
void	setJobSuccess() set job as success.
void	setVerificationEnabled(boolean enableVerification) Enable verification.
Journal.JournalEntry	toJournalEntry()
String	toString()
void	updateBandwidth(OptionalLong bandwidth) Update bandwidth.

Methods inherited from class alluxio.master.job.AbstractJob

getEndTime, getJobId, getJobState, getWorkerAssignPolicy, initializeJob, isDone, isRunning, setEndTime, setJobState, setStartTime, setWorkerAssignPolicy

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

TYPE

public static final String TYPE

See Also:

Constant Field Values

QUALIFIED_FILE_FILTER

public static final java.util.function.Predicate<FileInfo> QUALIFIED_FILE_FILTER

JOB_LOAD_SUCCESS

public static final com.codahale.metrics.Counter JOB_LOAD_SUCCESS

JOB_LOAD_FAIL

public static final com.codahale.metrics.Counter JOB_LOAD_FAIL

JOB_LOAD_BLOCK_COUNT

public static final com.codahale.metrics.Counter JOB_LOAD_BLOCK_COUNT

JOB_LOAD_BLOCK_FAIL

public static final com.codahale.metrics.Counter JOB_LOAD_BLOCK_FAIL

JOB_LOAD_BLOCK_SIZE

public static final com.codahale.metrics.Counter JOB_LOAD_BLOCK_SIZE

JOB_LOAD_RATE

public static final com.codahale.metrics.Meter JOB_LOAD_RATE

Constructor Detail

LoadJob

public LoadJob(String path,
               String user,
               OptionalLong bandwidth,
               FileIterable fileIterator)

Constructor.

Parameters:

path - file path

user - user for authentication

bandwidth - bandwidth

fileIterator - file iterator

LoadJob

public LoadJob(String path,
               Optional<String> user,
               String jobId,
               OptionalLong bandwidth,
               boolean usePartialListing,
               boolean verificationEnabled,
               Iterable<FileInfo> fileIterable)

Constructor.

Parameters:

path - file path

user - user for authentication

jobId - job identifier

bandwidth - bandwidth

usePartialListing - whether to use partial listing

verificationEnabled - whether to verify the job after loaded

fileIterable - file iterable

Method Detail

getPath

public String getPath()

Get load file path.

Returns:

file path

getDescription

public JobDescription getDescription()

Returns:

the job description, which is used to identify the intention of the job. We don't allow multiple jobs with the same description to be run on the scheduler at the same time. When submitting a job when there is already a job with the same description, scheduler will update the job instead of submitting a new job.

getBandwidth

public OptionalLong getBandwidth()

Get bandwidth.

Returns:

the allocated bandwidth

updateBandwidth

public void updateBandwidth(OptionalLong bandwidth)

Update bandwidth.

Parameters:

bandwidth - new bandwidth

isVerificationEnabled

public boolean isVerificationEnabled()

Is verification enabled.

Returns:

whether verification is enabled

setVerificationEnabled

public void setVerificationEnabled(boolean enableVerification)

Enable verification.

Parameters:

enableVerification - whether to enable verification

failJob

public void failJob(AlluxioRuntimeException reason)

Set load state to FAILED with given reason.

Parameters:

reason - failure exception

setJobSuccess

public void setJobSuccess()

Description copied from interface: Job

set job as success.

addLoadedBytes

public void addLoadedBytes(long bytes)

Add bytes to total loaded bytes.

Parameters:

bytes - bytes to be added to total

getProgress

public String getProgress(JobProgressReportFormat format,
                          boolean verbose)

Description copied from interface: Job

Get job progress.

Parameters:

format - progress report format

verbose - whether to include detailed information

Returns:

job progress report

getCurrentBlockCount

public long getCurrentBlockCount()

Get the processed block count in the current loading pass.

Returns:

current block count

equals

public boolean equals(Object o)

Overrides:

equals in class Object

hashCode

public int hashCode()

Overrides:

hashCode in class Object

isHealthy

public boolean isHealthy()

Description copied from interface: Job

Check whether the job is healthy.

Returns:

true if the job is healthy, false if not

isCurrentPassDone

public boolean isCurrentPassDone()

Description copied from interface: Job

Check whether the current pass is finished.

Returns:

true if the current pass of job is finished, false if not

initiateVerification

public void initiateVerification()

Description copied from interface: Job

Initiate a verification pass. This will re-list the directory and find any unfinished files / tasks and try to execute them again.

getNextTasks

public List<LoadJob.LoadTask> getNextTasks(Collection<WorkerInfo> workers)

get next load task.

Parameters:

workers - list of available workers to schedule task on

Returns:

the next task to run. If there is no task to run, return empty

onTaskSubmitFailure

public void onTaskSubmitFailure(Task<?> task)

Description copied from interface: Job

Define how to process task that gets rejected when scheduler tried to kick off.

Specified by:

onTaskSubmitFailure in interface Job<LoadJob.LoadTask>

Overrides:

onTaskSubmitFailure in class AbstractJob<LoadJob.LoadTask>

getNextBatchBlocks

public List<Block> getNextBatchBlocks(int count)

Get next batch of blocks.

Parameters:

count - number of blocks

Returns:

list of blocks

addBlockToRetry

public boolean addBlockToRetry(Block block)

Add a block to retry later.

Parameters:

block - the block that failed to load thus needing retry

Returns:

whether the block is successfully added

addBlockFailure

public void addBlockFailure(Block block,
                            String message,
                            int code)

Add a block to failure summary.

Parameters:

block - the block that failed to load and cannot be retried

message - failure message

code - status code for exception

toString

public String toString()

Overrides:

toString in class Object

toJournalEntry

public Journal.JournalEntry toJournalEntry()

Returns:

job journal entry

getDurationInSec

public long getDurationInSec()

Get duration in seconds.

Returns:

job duration in seconds

processResponse

public boolean processResponse(LoadJob.LoadTask loadTask)

Description copied from interface: Job

process task result.

Parameters:

loadTask - task containing result future

Returns:

success if successfully process task result, otherwise return failure

hasFailure

public boolean hasFailure()

Returns:

whether the job has failed tasks

needVerification

public boolean needVerification()

Is verification enabled.

Returns:

whether verification is enabled