alluxio.master.job

Class JobMaster

java.lang.Object

alluxio.master.AbstractMaster

alluxio.master.job.JobMaster

All Implemented Interfaces:

Checkpointed, Journaled, JournalEntryIterable, NoopJournaled, Master, Server<Boolean>

@ThreadSafe
public class JobMaster
extends AbstractMaster
implements NoopJournaled

The master that handles all job managing operations.

Field Summary

Fields inherited from class alluxio.master.AbstractMaster

mClock, mMasterContext

Constructor Summary

Constructors

Constructor and Description
JobMaster(MasterContext masterContext, FileSystem filesystem, FileSystemContext fsContext, UfsManager ufsManager) Creates a new instance of JobMaster.

Method Summary

Modifier and Type	Method and Description
void	cancel(long jobId) Cancels a job.
List<JobInfo>	failed(int limit, long before, long after)
List<JobWorkerHealth>	getAllWorkerHealth()
String	getName()
long	getNewJobId()
Map<ServiceType,GrpcService>	getServices()
JobInfo	getStatus(long jobId) Gets information of the given job id (verbose = True).
JobInfo	getStatus(long jobId, boolean verbose) Gets information of the given job id.
JobServiceSummary	getSummary() Gets summary of the job service.
List<WorkerInfo>	getWorkerInfoList()
List<Long>	list(ListAllPOptions options)
List<JobInfo>	listDetailed()
long	registerWorker(WorkerNetAddress workerNetAddress) Returns a worker id for the given worker.
long	run(JobConfig jobConfig) Runs a job with the given configuration.
void	run(JobConfig jobConfig, long jobId) Runs a job with the given configuration and job id.
void	setTaskPoolSize(int taskPoolSize)
void	start(Boolean isLeader) Starts the Alluxio server.
void	stop() Stops the Alluxio server.
List<JobCommand>	workerHeartbeat(JobWorkerHealth jobWorkerHealth, List<TaskInfo> taskInfoList) Updates the tasks' status when a worker periodically heartbeats with the master, and sends the commands for the worker to execute.

Methods inherited from class alluxio.master.AbstractMaster

close, createJournalContext, getDependencies, getExecutorService, getMasterContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface alluxio.master.journal.NoopJournaled

getCheckpointName, getJournalEntryIterator, processJournalEntry, resetState, restoreFromCheckpoint, writeToCheckpoint

Methods inherited from interface alluxio.master.journal.Journaled

applyAndJournal

Constructor Detail

JobMaster

public JobMaster(MasterContext masterContext,
                 FileSystem filesystem,
                 FileSystemContext fsContext,
                 UfsManager ufsManager)

Creates a new instance of JobMaster.

Parameters:

masterContext - the context for Alluxio master

filesystem - the Alluxio filesystem client the job master uses to communicate

fsContext - the filesystem client's underlying context

ufsManager - the ufs manager

Method Detail

getNewJobId

public long getNewJobId()

Returns:

new job id

start

public void start(Boolean isLeader)
           throws IOException

Description copied from interface: Server

Starts the Alluxio server.

Specified by:

start in interface Server<Boolean>

Overrides:

start in class AbstractMaster

Parameters:

isLeader - the start options

Throws:

IOException

stop

public void stop()
          throws IOException

Description copied from interface: Server

Stops the Alluxio server. Here, anything created or started in #start(T) should be cleaned up and shutdown.

Specified by:

stop in interface Server<Boolean>

Overrides:

stop in class AbstractMaster

Throws:

IOException

getServices

public Map<ServiceType,GrpcService> getServices()

Specified by:

getServices in interface Server<Boolean>

Returns:

a map from service names to gRPC serviced that serve RPCs for this server

getName

public String getName()

Specified by:

getName in interface Server<Boolean>

Returns:

the server's name

run

public long run(JobConfig jobConfig)
         throws JobDoesNotExistException,
                ResourceExhaustedException

Runs a job with the given configuration.

Parameters:

jobConfig - the job configuration

Returns:

the job id tracking the progress

Throws:

JobDoesNotExistException - when the job doesn't exist

ResourceExhaustedException - if the job master is too busy to run the job

run

public void run(JobConfig jobConfig,
                long jobId)
         throws JobDoesNotExistException,
                ResourceExhaustedException

Runs a job with the given configuration and job id.

Parameters:

jobConfig - the job configuration

jobId - the job id

Throws:

JobDoesNotExistException - when the job doesn't exist

ResourceExhaustedException - if the job master is too busy to run the job

cancel

public void cancel(long jobId)
            throws JobDoesNotExistException

Cancels a job.

Parameters:

jobId - the id of the job

Throws:

JobDoesNotExistException - when the job does not exist

list

public List<Long> list(ListAllPOptions options)

Parameters:

options - listing options

Returns:

list of all job ids

listDetailed

public List<JobInfo> listDetailed()

Returns:

list of all job infos

failed

public List<JobInfo> failed(int limit,
                            long before,
                            long after)

Parameters:

limit - maximum number of jobInfos to return

before - filters out on or after this timestamp (in ms) (-1 to disable)

after - filter out on or before this timestamp (in ms) (-1 to disable)

Returns:

list of all failed job infos ordered by when it failed (recently failed first)

getStatus

public JobInfo getStatus(long jobId)
                  throws JobDoesNotExistException

Gets information of the given job id (verbose = True).

Parameters:

jobId - the id of the job

Returns:

the job information

Throws:

JobDoesNotExistException - if the job does not exist

getStatus

public JobInfo getStatus(long jobId,
                         boolean verbose)
                  throws JobDoesNotExistException

Gets information of the given job id.

Parameters:

jobId - the id of the job

verbose - whether the job info should be verbose

Returns:

the job information

Throws:

JobDoesNotExistException - if the job does not exist

getSummary

public JobServiceSummary getSummary()

Gets summary of the job service.

Returns:

JobServiceSummary

getAllWorkerHealth

public List<JobWorkerHealth> getAllWorkerHealth()

Returns:

health metrics for each of the job workers

registerWorker

public long registerWorker(WorkerNetAddress workerNetAddress)

Returns a worker id for the given worker.

Parameters:

workerNetAddress - the worker WorkerNetAddress

Returns:

the worker id for this worker

getWorkerInfoList

public List<WorkerInfo> getWorkerInfoList()

Returns:

a list of WorkerInfo objects representing the workers in Alluxio

setTaskPoolSize

public void setTaskPoolSize(int taskPoolSize)

Parameters:

taskPoolSize - the task pool size for the job workers

workerHeartbeat

public List<JobCommand> workerHeartbeat(JobWorkerHealth jobWorkerHealth,
                                        List<TaskInfo> taskInfoList)
                                 throws ResourceExhaustedException

Updates the tasks' status when a worker periodically heartbeats with the master, and sends the commands for the worker to execute.

Parameters:

jobWorkerHealth - the job worker health info

taskInfoList - the list of the task information

Returns:

the list of JobCommand to the worker

Throws:

ResourceExhaustedException