alluxio.master.scheduler

Class Scheduler

java.lang.Object

alluxio.master.scheduler.Scheduler

@ThreadSafe
public final class Scheduler
extends Object

The Scheduler which controls jobs. It is responsible for managing active workers, updating jobs and update job information to job meta store. The workflow is: 1. Submit a job to the scheduler. 2. The scheduler will pull the task from the job and assign the task to a worker. 3. The worker will execute the task and report the result to the job. 4. The job will update the progress. And schedule the next task if the job is not done. 5. One worker would have one task running for one job description at a time.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	Scheduler.BoundedPriorityBlockingQueue<E> Bounded priority queue impl.
static class	Scheduler.SchedulerStats Job/Tasks stats.
class	Scheduler.WorkerInfoHub Worker information hub.
static class	Scheduler.WorkerInfoIdentity Util class here for tracking unique identity of a worker as WorkerInfo class uses constantly changing field such as mLastContactSec for equals(), which can't be served as key class in map.

Constructor Summary

Constructors

Constructor and Description
Scheduler(FileSystemContext fsCtx, WorkerProvider workerProvider, JobMetaStore jobMetaStore) Constructor.

Method Summary

Modifier and Type	Method and Description
void	cleanupStaleJob() Removes all finished jobs outside the retention time.
Set<WorkerInfo>	getActiveWorkers() Get active workers.
static Scheduler	getInstance() Get the singleton instance of Scheduler.
JobMetaStore	getJobMetaStore() Get job meta store.
String	getJobProgress(JobDescription jobDescription, JobProgressReportFormat format, boolean verbose) Get the job's progress report.
Map<JobDescription,Job<?>>	getJobs() Get jobs.
JobState	getJobState(JobDescription jobDescription) Get the job's state.
Scheduler.WorkerInfoHub	getWorkerInfoHub() Get the workerinfo hub.
Scheduler.SchedulerStats	printJobsStatus() Print job status.
void	start() Start scheduler.
void	stop() Stop scheduler.
boolean	stopJob(JobDescription jobDescription) Stop a job.
boolean	submitJob(Job<?> job) Submit a job.
void	updateWorkers() Update workers.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Scheduler

public Scheduler(FileSystemContext fsCtx,
                 WorkerProvider workerProvider,
                 JobMetaStore jobMetaStore)

Constructor.

Parameters:

fsCtx - file system context

workerProvider - workerProvider

jobMetaStore - jobMetaStore

Method Detail

getInstance

@Nullable
public static Scheduler getInstance()

Get the singleton instance of Scheduler. getInstance won't be called before constructor.

Returns:

Scheduler instance

start

public void start()

Start scheduler.

updateWorkers

public void updateWorkers()

Update workers.

stop

public void stop()

Stop scheduler.

submitJob

public boolean submitJob(Job<?> job)

Submit a job.

Parameters:

job - the job

Returns:

true if the job is new, false if the job has already been submitted

Throws:

ResourceExhaustedRuntimeException - if the job cannot be submitted because the scheduler is at capacity

UnavailableRuntimeException - if the job cannot be submitted because the meta store is not ready

stopJob

public boolean stopJob(JobDescription jobDescription)

Stop a job.

Parameters:

jobDescription - job identifier

Returns:

true if the job is stopped, false if the job does not exist or has already finished

getJobProgress

public String getJobProgress(JobDescription jobDescription,
                             JobProgressReportFormat format,
                             boolean verbose)

Get the job's progress report.

Parameters:

jobDescription - job identifier

format - progress report format

verbose - whether to include details on failed files and failures

Returns:

the progress report

Throws:

NotFoundRuntimeException - if the job does not exist

AlluxioRuntimeException - if any other Alluxio exception occurs

getJobState

public JobState getJobState(JobDescription jobDescription)

Get the job's state.

Parameters:

jobDescription - job identifier

Returns:

the job state

Throws:

NotFoundRuntimeException - if the job does not exist

getActiveWorkers

public Set<WorkerInfo> getActiveWorkers()

Get active workers.

Returns:

active workers

cleanupStaleJob

public void cleanupStaleJob()

Removes all finished jobs outside the retention time.

getJobs

public Map<JobDescription,Job<?>> getJobs()

Get jobs.

Returns:

jobs

getWorkerInfoHub

public Scheduler.WorkerInfoHub getWorkerInfoHub()

Get the workerinfo hub.

Returns:

worker info hub

getJobMetaStore

public JobMetaStore getJobMetaStore()

Get job meta store.

Returns:

jobmetastore

printJobsStatus

public Scheduler.SchedulerStats printJobsStatus()

Print job status.

Returns:

SchedulerStats