alluxio.master.job

Class DoraLoadJob

java.lang.Object

alluxio.master.job.AbstractJob<DoraLoadJob.DoraLoadTask>

alluxio.master.job.DoraLoadJob

All Implemented Interfaces:

Job<DoraLoadJob.DoraLoadTask>

@NotThreadSafe
public class DoraLoadJob
extends AbstractJob<DoraLoadJob.DoraLoadTask>

Load job that loads a file or a directory into Alluxio. This class should only be manipulated from the scheduler thread in Scheduler thus the state changing functions are not thread safe. TODO() as task within this class is running on multithreaded context, make thread unsafe places to be thread safe in future.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	DoraLoadJob.DoraLoadTask Dora load task.

Field Summary

Fields

Modifier and Type	Field and Description
static com.codahale.metrics.Counter	JOB_LOAD_FAIL
static com.codahale.metrics.Counter	JOB_LOAD_FILE_COUNT
static com.codahale.metrics.Counter	JOB_LOAD_FILE_FAIL
static com.codahale.metrics.Counter	JOB_LOAD_FILE_SIZE
static com.codahale.metrics.Meter	JOB_LOAD_RATE
static com.codahale.metrics.Counter	JOB_LOAD_SUCCESS
static String	TYPE

Fields inherited from class alluxio.master.job.AbstractJob

mEndTime, mJobId, mRetryTaskList, mStartTime, mState, mTaskIdGenerator, mUser, mWorkerAssignPolicy

Constructor Summary

Constructors

Constructor and Description
DoraLoadJob(String path, Optional<String> user, String jobId, OptionalLong bandwidth, boolean usePartialListing, boolean verificationEnabled, boolean loadMetadataOnly, boolean skipIfExists) Constructor.

Method Summary

Modifier and Type	Method and Description
void	addFileFailure(String fileUfsPath, String type, String message) Add failed files.
boolean	addFilesToRetry(String path, String type, String message) Add files to retry.
void	addLoadedBytes(long bytes) Add bytes to total loaded bytes.
boolean	equals(Object o)
void	failJob(AlluxioRuntimeException reason) Set load state to FAILED with given reason.
OptionalLong	getBandwidth() Get bandwidth.
JobDescription	getDescription()
long	getDurationInSec() Get duration in seconds.
List<DoraLoadJob.DoraLoadTask>	getNextTasks(Collection<WorkerInfo> workers)
String	getPath() Get load file path.
String	getProgress(JobProgressReportFormat format, boolean verbose) Get job progress.
boolean	hasFailure()
int	hashCode()
void	initiateVerification() Initiate a verification pass.
boolean	isCurrentPassDone() Check whether the current pass is finished.
boolean	isHealthy() Check whether the job is healthy.
boolean	needVerification() Is verification enabled.
boolean	processResponse(DoraLoadJob.DoraLoadTask doraLoadTask) process task result.
void	setJobSuccess() set job as success.
Journal.JournalEntry	toJournalEntry()
String	toString()
void	updateBandwidth(OptionalLong bandwidth) Update bandwidth.

Methods inherited from class alluxio.master.job.AbstractJob

getEndTime, getJobId, getJobState, getWorkerAssignPolicy, initializeJob, isDone, isRunning, onTaskSubmitFailure, setEndTime, setJobState, setStartTime, setWorkerAssignPolicy

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

TYPE

public static final String TYPE

See Also:

Constant Field Values

JOB_LOAD_SUCCESS

public static final com.codahale.metrics.Counter JOB_LOAD_SUCCESS

JOB_LOAD_FAIL

public static final com.codahale.metrics.Counter JOB_LOAD_FAIL

JOB_LOAD_FILE_COUNT

public static final com.codahale.metrics.Counter JOB_LOAD_FILE_COUNT

JOB_LOAD_FILE_FAIL

public static final com.codahale.metrics.Counter JOB_LOAD_FILE_FAIL

JOB_LOAD_FILE_SIZE

public static final com.codahale.metrics.Counter JOB_LOAD_FILE_SIZE

JOB_LOAD_RATE

public static final com.codahale.metrics.Meter JOB_LOAD_RATE

Constructor Detail

DoraLoadJob

public DoraLoadJob(String path,
                   Optional<String> user,
                   String jobId,
                   OptionalLong bandwidth,
                   boolean usePartialListing,
                   boolean verificationEnabled,
                   boolean loadMetadataOnly,
                   boolean skipIfExists)

Constructor.

Parameters:

path - file path

user - user for authentication

jobId - job identifier

bandwidth - bandwidth

usePartialListing - whether to use partial listing

verificationEnabled - whether to verify the job after loaded

loadMetadataOnly - if set to true, only metadata will be loaded without loading file data

skipIfExists - skip if exists

Method Detail

getPath

public String getPath()

Get load file path.

Returns:

file path

getDescription

public JobDescription getDescription()

Returns:

the job description, which is used to identify the intention of the job. We don't allow multiple jobs with the same description to be run on the scheduler at the same time. When submitting a job when there is already a job with the same description, scheduler will update the job instead of submitting a new job.

getBandwidth

public OptionalLong getBandwidth()

Get bandwidth.

Returns:

the allocated bandwidth

updateBandwidth

public void updateBandwidth(OptionalLong bandwidth)

Update bandwidth.

Parameters:

bandwidth - new bandwidth

failJob

public void failJob(AlluxioRuntimeException reason)

Set load state to FAILED with given reason.

Parameters:

reason - failure exception

setJobSuccess

public void setJobSuccess()

Description copied from interface: Job

set job as success.

addLoadedBytes

public void addLoadedBytes(long bytes)

Add bytes to total loaded bytes.

Parameters:

bytes - bytes to be added to total

addFilesToRetry

public boolean addFilesToRetry(String path,
                               String type,
                               String message)

Add files to retry.

Parameters:

path - the file path

type - the error type

message - the error message

Returns:

true

addFileFailure

public void addFileFailure(String fileUfsPath,
                           String type,
                           String message)

Add failed files.

Parameters:

fileUfsPath -

message -

type -

getProgress

public String getProgress(JobProgressReportFormat format,
                          boolean verbose)

Description copied from interface: Job

Get job progress.

Parameters:

format - progress report format

verbose - whether to include detailed information

Returns:

job progress report

equals

public boolean equals(Object o)

Overrides:

equals in class Object

hashCode

public int hashCode()

Overrides:

hashCode in class Object

isHealthy

public boolean isHealthy()

Description copied from interface: Job

Check whether the job is healthy.

Returns:

true if the job is healthy, false if not

isCurrentPassDone

public boolean isCurrentPassDone()

Description copied from interface: Job

Check whether the current pass is finished.

Returns:

true if the current pass of job is finished, false if not

initiateVerification

public void initiateVerification()

Description copied from interface: Job

Initiate a verification pass. This will re-list the directory and find any unfinished files / tasks and try to execute them again.

getNextTasks

public List<DoraLoadJob.DoraLoadTask> getNextTasks(Collection<WorkerInfo> workers)

Parameters:

workers - blocker to worker

Returns:

the next task to run. If there is no more task to run, return empty

toString

public String toString()

Overrides:

toString in class Object

toJournalEntry

public Journal.JournalEntry toJournalEntry()

Returns:

job journal entry

getDurationInSec

public long getDurationInSec()

Get duration in seconds.

Returns:

job duration in seconds

processResponse

public boolean processResponse(DoraLoadJob.DoraLoadTask doraLoadTask)

Description copied from interface: Job

process task result.

Parameters:

doraLoadTask - task containing result future

Returns:

success if successfully process task result, otherwise return failure

hasFailure

public boolean hasFailure()

Returns:

whether the job has failed tasks

needVerification

public boolean needVerification()

Is verification enabled.

Returns:

whether verification is enabled