alluxio.master.file.loadmanager

Class LoadJob

java.lang.Object

alluxio.master.file.loadmanager.LoadJob

@NotThreadSafe
public class LoadJob
extends Object

This class should only be manipulated from the scheduler thread in LoadManager thus the state changing functions are not thread safe.

Constructor Summary

Constructors

Constructor and Description
LoadJob(String path, Optional<String> user, String jobId, OptionalLong bandwidth, boolean usePartialListing, boolean verificationEnabled) Constructor.
LoadJob(String path, String user, OptionalLong bandwidth) Constructor.

Method Summary

Modifier and Type	Method and Description
void	addBlockFailure(Block block, String message, int code) Add a block to failure summary.
boolean	addBlockToRetry(Block block) Add a block to retry later.
void	addLoadedBytes(long bytes) Add bytes to total loaded bytes.
boolean	equals(Object o)
void	failJob(AlluxioRuntimeException reason) Set load state to FAILED with given reason.
static LoadJob	fromJournalEntry(Job.LoadJobEntry loadJobEntry) Get journal entry of the job.
OptionalLong	getBandwidth() Get bandwidth.
int	getBatchSize() Get batch size.
long	getCurrentBlockCount() Get the processed block count in the current loading pass.
long	getDurationInSec() Get duration in seconds.
OptionalLong	getEndTime() Get end time.
String	getJobId() Get uniq tag.
LoadJobState	getJobState() Get load status.
List<Block>	getNextBatch(FileSystemMaster fileSystemMaster, int count) Get next batch of blocks.
String	getPath() Get load file path.
String	getProgress(LoadProgressReportFormat format, boolean verbose) Get load job progress.
long	getTotalBlockCount() Get the total processed block count for this job.
Optional<String>	getUser() Get user.
int	hashCode()
void	initiateVerification() Initiate a verification pass.
boolean	isCurrentLoadDone() Check whether the current loading pass is finished.
boolean	isDone() Check whether the load job is finished.
boolean	isHealthy() Check whether the load job is healthy.
boolean	isRunning() Check whether the load job is still running.
boolean	isVerificationEnabled() Is verification enabled.
void	setEndTime(long time) Update end time.
void	setJobState(LoadJobState state) Set load state.
void	setVerificationEnabled(boolean enableVerification) Enable verification.
Journal.JournalEntry	toJournalEntry()
String	toString()
void	updateBandwidth(OptionalLong bandwidth) Update bandwidth.

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

LoadJob

public LoadJob(String path,
               String user,
               OptionalLong bandwidth)

Constructor.

Parameters:

path - file path

user - user for authentication

bandwidth - bandwidth

LoadJob

public LoadJob(String path,
               Optional<String> user,
               String jobId,
               OptionalLong bandwidth,
               boolean usePartialListing,
               boolean verificationEnabled)

Constructor.

Parameters:

path - file path

user - user for authentication

jobId - job identifier

bandwidth - bandwidth

usePartialListing - whether to use partial listing

verificationEnabled - whether to verify the job after loaded

Method Detail

getPath

public String getPath()

Get load file path.

Returns:

file path

getUser

public Optional<String> getUser()

Get user.

Returns:

user

getEndTime

public OptionalLong getEndTime()

Get end time.

Returns:

end time

getBandwidth

public OptionalLong getBandwidth()

Get bandwidth.

Returns:

the allocated bandwidth

setEndTime

public void setEndTime(long time)

Update end time.

Parameters:

time - time in ms

updateBandwidth

public void updateBandwidth(OptionalLong bandwidth)

Update bandwidth.

Parameters:

bandwidth - new bandwidth

isVerificationEnabled

public boolean isVerificationEnabled()

Is verification enabled.

Returns:

whether verification is enabled

setVerificationEnabled

public void setVerificationEnabled(boolean enableVerification)

Enable verification.

Parameters:

enableVerification - whether to enable verification

getJobState

public LoadJobState getJobState()

Get load status.

Returns:

the load job's status

setJobState

public void setJobState(LoadJobState state)

Set load state.

Parameters:

state - new state

getJobId

public String getJobId()

Get uniq tag.

Returns:

the tag

failJob

public void failJob(AlluxioRuntimeException reason)

Set load state to FAILED with given reason.

Parameters:

reason - failure exception

getBatchSize

public int getBatchSize()

Get batch size.

Returns:

batch size

addLoadedBytes

public void addLoadedBytes(long bytes)

Add bytes to total loaded bytes.

Parameters:

bytes - bytes to be added to total

getProgress

public String getProgress(LoadProgressReportFormat format,
                          boolean verbose)

Get load job progress.

Parameters:

format - report format

verbose - whether to include error details in the report

Returns:

the load progress report

getCurrentBlockCount

public long getCurrentBlockCount()

Get the processed block count in the current loading pass.

Returns:

current block count

getTotalBlockCount

public long getTotalBlockCount()

Get the total processed block count for this job.

Returns:

total block count

equals

public boolean equals(Object o)

Overrides:

equals in class Object

hashCode

public int hashCode()

Overrides:

hashCode in class Object

isHealthy

public boolean isHealthy()

Check whether the load job is healthy.

Returns:

true if the load job is healthy, false if not

isRunning

public boolean isRunning()

Check whether the load job is still running.

Returns:

true if the load job is running, false if not

isDone

public boolean isDone()

Check whether the load job is finished.

Returns:

true if the load job is finished, false if not

isCurrentLoadDone

public boolean isCurrentLoadDone()

Check whether the current loading pass is finished.

Returns:

true if the load job is finished, false if not

initiateVerification

public void initiateVerification()

Initiate a verification pass. This will re-list the directory and find any unloaded files / blocks and try to load them again.

getNextBatch

public List<Block> getNextBatch(FileSystemMaster fileSystemMaster,
                                int count)

Get next batch of blocks.

Parameters:

fileSystemMaster - file system master to fetch file infos

count - number of blocks

Returns:

list of blocks

addBlockToRetry

public boolean addBlockToRetry(Block block)

Add a block to retry later.

Parameters:

block - the block that failed to load thus needing retry

Returns:

whether the block is successfully added

addBlockFailure

public void addBlockFailure(Block block,
                            String message,
                            int code)

Add a block to failure summary.

Parameters:

block - the block that failed to load and cannot be retried

message - failure message

code - status code for exception

toString

public String toString()

Overrides:

toString in class Object

toJournalEntry

public Journal.JournalEntry toJournalEntry()

Returns:

journal entry of job

fromJournalEntry

public static LoadJob fromJournalEntry(Job.LoadJobEntry loadJobEntry)

Get journal entry of the job.

Parameters:

loadJobEntry - journal entry

Returns:

journal entry of the job

getDurationInSec

public long getDurationInSec()

Get duration in seconds.

Returns:

job duration in seconds