alluxio.master.file.loadmanager

Class LoadManager

java.lang.Object

alluxio.master.file.loadmanager.LoadManager

All Implemented Interfaces:

Checkpointed, Journaled, JournalEntryIterable

@ThreadSafe
public final class LoadManager
extends Object
implements Journaled

The Load manager which controls load operations. It's not thread-safe since start and stop method is not thread-safe. But we should only have one thread call these two method.

Field Summary

Fields

Modifier and Type	Field and Description
static com.codahale.metrics.Counter	JOB_LOAD_BLOCK_COUNT
static com.codahale.metrics.Counter	JOB_LOAD_BLOCK_FAIL
static com.codahale.metrics.Counter	JOB_LOAD_BLOCK_SIZE
static com.codahale.metrics.Counter	JOB_LOAD_FAIL
static com.codahale.metrics.Meter	JOB_LOAD_RATE
static com.codahale.metrics.Counter	JOB_LOAD_SUCCESS

Constructor Summary

Constructors

Constructor and Description
LoadManager(FileSystemMaster fileSystemMaster) Constructor.
LoadManager(FileSystemMaster fileSystemMaster, FileSystemContext context) Constructor.

Method Summary

Modifier and Type	Method and Description
void	cleanupStaleJob() Removes all finished jobs outside the retention time.
Map<WorkerInfo,CloseableResource<BlockWorkerClient>>	getActiveWorkers() Get active workers.
CheckpointName	getCheckpointName()
CloseableIterator<Journal.JournalEntry>	getJournalEntryIterator()
Map<String,LoadJob>	getLoadJobs() Get load jobs.
String	getLoadProgress(String loadPath, LoadProgressReportFormat format, boolean verbose) Get the load job's progress report.
boolean	processJournalEntry(Journal.JournalEntry entry) Attempts to apply a journal entry.
void	resetState() Resets the object's journaled state.
void	start() Start load manager.
void	stop() Stop load manager.
boolean	stopLoad(String loadPath) Stop a load job.
boolean	submitLoad(LoadJob loadJob) Submit a load job.
boolean	submitLoad(String loadPath, OptionalLong bandwidth, boolean usePartialListing, boolean verificationEnabled) Submit a load job.
void	updateWorkers() Refresh active workers.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface alluxio.master.journal.Journaled

applyAndJournal, restoreFromCheckpoint, writeToCheckpoint

Field Detail

JOB_LOAD_SUCCESS

public static final com.codahale.metrics.Counter JOB_LOAD_SUCCESS

JOB_LOAD_FAIL

public static final com.codahale.metrics.Counter JOB_LOAD_FAIL

JOB_LOAD_BLOCK_COUNT

public static final com.codahale.metrics.Counter JOB_LOAD_BLOCK_COUNT

JOB_LOAD_BLOCK_FAIL

public static final com.codahale.metrics.Counter JOB_LOAD_BLOCK_FAIL

JOB_LOAD_BLOCK_SIZE

public static final com.codahale.metrics.Counter JOB_LOAD_BLOCK_SIZE

JOB_LOAD_RATE

public static final com.codahale.metrics.Meter JOB_LOAD_RATE

Constructor Detail

LoadManager

public LoadManager(FileSystemMaster fileSystemMaster)

Constructor.

Parameters:

fileSystemMaster - fileSystemMaster

LoadManager

public LoadManager(FileSystemMaster fileSystemMaster,
                   FileSystemContext context)

Constructor.

Parameters:

fileSystemMaster - fileSystemMaster

context - fileSystemContext

Method Detail

start

public void start()

Start load manager.

stop

public void stop()

Stop load manager.

submitLoad

public boolean submitLoad(String loadPath,
                          OptionalLong bandwidth,
                          boolean usePartialListing,
                          boolean verificationEnabled)

Submit a load job.

Parameters:

loadPath - alluxio directory path to load into Alluxio

bandwidth - bandwidth allocated to this load

usePartialListing - whether to use partial listing or not

verificationEnabled - whether to run verification step or not

Returns:

true if the job is new, false if the job has already been submitted

submitLoad

public boolean submitLoad(LoadJob loadJob)

Submit a load job.

Parameters:

loadJob - the load job

Returns:

true if the job is new, false if the job has already been submitted

stopLoad

public boolean stopLoad(String loadPath)

Stop a load job.

Parameters:

loadPath - alluxio directory path to load into Alluxio

Returns:

true if the job is stopped, false if the job does not exist or has already finished

getLoadProgress

public String getLoadProgress(String loadPath,
                              LoadProgressReportFormat format,
                              boolean verbose)

Get the load job's progress report.

Parameters:

loadPath - alluxio directory path of the load job

format - progress report format

verbose - whether to include details on failed files and failures

Returns:

the progress report

getActiveWorkers

public Map<WorkerInfo,CloseableResource<BlockWorkerClient>> getActiveWorkers()

Get active workers.

Returns:

active workers

cleanupStaleJob

public void cleanupStaleJob()

Removes all finished jobs outside the retention time.

updateWorkers

public void updateWorkers()

Refresh active workers.

getLoadJobs

public Map<String,LoadJob> getLoadJobs()

Get load jobs.

Returns:

load jobs

getJournalEntryIterator

public CloseableIterator<Journal.JournalEntry> getJournalEntryIterator()

Specified by:

getJournalEntryIterator in interface JournalEntryIterable

Returns:

an CloseableIterator that iterates all the journal entries

processJournalEntry

public boolean processJournalEntry(Journal.JournalEntry entry)

Description copied from interface: Journaled

Attempts to apply a journal entry.

Specified by:

processJournalEntry in interface Journaled

Parameters:

entry - the entry to apply

Returns:

whether the entry type is supported by this journaled object

resetState

public void resetState()

Description copied from interface: Journaled

Resets the object's journaled state.

Specified by:

resetState in interface Journaled

getCheckpointName

public CheckpointName getCheckpointName()

Specified by:

getCheckpointName in interface Checkpointed

Returns:

a name for this checkpointed class