alluxio.master.job

Class JobMaster

java.lang.Object

alluxio.master.AbstractMaster

alluxio.master.job.JobMaster

All Implemented Interfaces:

Checkpointed, Journaled, JournalEntryIterable, NoopJournaled, Master, Server<Boolean>

@ThreadSafe
public class JobMaster
extends AbstractMaster
implements NoopJournaled

The master that handles all job managing operations.

Field Summary

Fields inherited from class alluxio.master.AbstractMaster

mClock, mJournal, mMasterContext

Constructor Summary

Constructors

Constructor and Description
JobMaster(MasterContext masterContext, FileSystem filesystem, FileSystemContext fsContext, UfsManager ufsManager) Creates a new instance of JobMaster.

Method Summary

Modifier and Type	Method and Description
void	cancel(long jobId) Cancels a job.
List<JobInfo>	failed(int limit, long before, long after)
Set<String>	getAllFailedPaths()
List<JobMasterStatus>	getAllJobMasterStatus() Lists the status of all job masters in the cluster.
List<JobWorkerHealth>	getAllWorkerHealth()
Status	getCmdStatus(long jobControlId) Get command status.
CmdStatusBlock	getCmdStatusDetailed(long jobControlId)
Set<String>	getFailedPaths(long jobControlId)
long	getMasterId(Address address) Allocates an ID for the job master from the target address.
String	getName()
long	getNewJobId()
Map<ServiceType,GrpcService>	getServices()
JobInfo	getStatus(long jobId) Gets information of the given job id (verbose = True).
JobInfo	getStatus(long jobId, boolean verbose) Gets information of the given job id.
JobServiceSummary	getSummary() Gets summary of the job service.
List<WorkerInfo>	getWorkerInfoList()
JobMasterMetaCommand	jobMasterHeartbeat(long masterId, JobMasterHeartbeatPOptions options) Handles a heartbeat from a standby job master.
void	jobMasterRegister(long masterId, RegisterJobMasterPOptions options) Handles the register request from a standby job master.
List<Long>	list(ListAllPOptions options)
List<Long>	listCmds(ListAllPOptions options)
List<JobInfo>	listDetailed()
long	registerWorker(WorkerNetAddress workerNetAddress, BuildVersion version) Returns a worker id for the given worker.
long	run(JobConfig jobConfig) Runs a job with the given configuration.
void	run(JobConfig jobConfig, long jobId) Runs a job with the given configuration and job id.
void	setTaskPoolSize(int taskPoolSize)
void	start(Boolean isLeader) Starts the Alluxio server.
void	stop() Stops the Alluxio server.
long	submit(CmdConfig cmdConfig) Submit a job with the given configuration.
List<JobCommand>	workerHeartbeat(JobWorkerHealth jobWorkerHealth, List<TaskInfo> taskInfoList) Updates the tasks' status when a worker periodically heartbeats with the master, and sends the commands for the worker to execute.

Methods inherited from class alluxio.master.AbstractMaster

close, createJournalContext, getDependencies, getExecutorService, getMasterContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface alluxio.master.journal.NoopJournaled

getCheckpointName, getJournalEntryIterator, processJournalEntry, resetState, restoreFromCheckpoint, restoreFromCheckpoint, writeToCheckpoint, writeToCheckpoint

Methods inherited from interface alluxio.master.journal.Journaled

applyAndJournal

Methods inherited from interface alluxio.master.Master

getStandbyServices

Constructor Detail

JobMaster

public JobMaster(MasterContext masterContext,
                 FileSystem filesystem,
                 FileSystemContext fsContext,
                 UfsManager ufsManager)

Creates a new instance of JobMaster.

Parameters:

masterContext - the context for Alluxio master

filesystem - the Alluxio filesystem client the job master uses to communicate

fsContext - the filesystem client's underlying context

ufsManager - the ufs manager

Method Detail

getNewJobId

public long getNewJobId()

Returns:

new job id

start

public void start(Boolean isLeader)
           throws IOException

Description copied from interface: Server

Starts the Alluxio server.

Specified by:

start in interface Server<Boolean>

Overrides:

start in class AbstractMaster

Parameters:

isLeader - the start options

Throws:

IOException

stop

public void stop()
          throws IOException

Description copied from interface: Server

Stops the Alluxio server. Here, anything created or started in #start(T) should be cleaned up and shutdown.

Specified by:

stop in interface Server<Boolean>

Overrides:

stop in class AbstractMaster

Throws:

IOException

getServices

public Map<ServiceType,GrpcService> getServices()

Specified by:

getServices in interface Server<Boolean>

Returns:

a map from service names to gRPC serviced that serve RPCs for this server

getName

public String getName()

Specified by:

getName in interface Server<Boolean>

Returns:

the server's name

run

public long run(JobConfig jobConfig)
         throws JobDoesNotExistException,
                ResourceExhaustedException

Runs a job with the given configuration.

Parameters:

jobConfig - the job configuration

Returns:

the job id tracking the progress

Throws:

JobDoesNotExistException - when the job doesn't exist

ResourceExhaustedException - if the job master is too busy to run the job

run

public void run(JobConfig jobConfig,
                long jobId)
         throws JobDoesNotExistException,
                ResourceExhaustedException

Runs a job with the given configuration and job id.

Parameters:

jobConfig - the job configuration

jobId - the job id

Throws:

JobDoesNotExistException - when the job doesn't exist

ResourceExhaustedException - if the job master is too busy to run the job

submit

public long submit(CmdConfig cmdConfig)
            throws JobDoesNotExistException,
                   IOException

Submit a job with the given configuration.

Parameters:

cmdConfig - the CMD configuration

Returns:

the job control id tracking the progress

Throws:

JobDoesNotExistException - when the job doesn't exist

ResourceExhaustedException - if the job master is too busy to run the job

IOException

cancel

public void cancel(long jobId)
            throws JobDoesNotExistException

Cancels a job.

Parameters:

jobId - the id of the job

Throws:

JobDoesNotExistException - when the job does not exist

getCmdStatus

public Status getCmdStatus(long jobControlId)
                    throws JobDoesNotExistException

Get command status.

Parameters:

jobControlId -

Returns:

status of a distributed commmand

Throws:

JobDoesNotExistException

list

public List<Long> list(ListAllPOptions options)

Parameters:

options - listing options

Returns:

list of all job ids

listCmds

public List<Long> listCmds(ListAllPOptions options)
                    throws JobDoesNotExistException

Parameters:

options - listing options (using existing options)

Returns:

list of all command ids

Throws:

JobDoesNotExistException

getCmdStatusDetailed

public CmdStatusBlock getCmdStatusDetailed(long jobControlId)
                                    throws JobDoesNotExistException

Parameters:

jobControlId - job control ID of a command

Returns:

get a detailed status information for a command

Throws:

JobDoesNotExistException

getAllFailedPaths

public Set<String> getAllFailedPaths()

Returns:

all failed paths

getFailedPaths

public Set<String> getFailedPaths(long jobControlId)
                           throws JobDoesNotExistException

Parameters:

jobControlId - job control id

Returns:

get failed paths for a command

Throws:

JobDoesNotExistException

listDetailed

public List<JobInfo> listDetailed()

Returns:

list of all job infos

failed

public List<JobInfo> failed(int limit,
                            long before,
                            long after)

Parameters:

limit - maximum number of jobInfos to return

before - filters out on or after this timestamp (in ms) (-1 to disable)

after - filter out on or before this timestamp (in ms) (-1 to disable)

Returns:

list of all failed job infos ordered by when it failed (recently failed first)

getStatus

public JobInfo getStatus(long jobId)
                  throws JobDoesNotExistException

Gets information of the given job id (verbose = True).

Parameters:

jobId - the id of the job

Returns:

the job information

Throws:

JobDoesNotExistException - if the job does not exist

getStatus

public JobInfo getStatus(long jobId,
                         boolean verbose)
                  throws JobDoesNotExistException

Gets information of the given job id.

Parameters:

jobId - the id of the job

verbose - whether the job info should be verbose

Returns:

the job information

Throws:

JobDoesNotExistException - if the job does not exist

getSummary

public JobServiceSummary getSummary()

Gets summary of the job service.

Returns:

JobServiceSummary

getAllWorkerHealth

public List<JobWorkerHealth> getAllWorkerHealth()

Returns:

health metrics for each of the job workers

getAllJobMasterStatus

public List<JobMasterStatus> getAllJobMasterStatus()

Lists the status of all job masters in the cluster.

Returns:

the list

registerWorker

public long registerWorker(WorkerNetAddress workerNetAddress,
                           BuildVersion version)

Returns a worker id for the given worker.

Parameters:

workerNetAddress - the worker WorkerNetAddress

version - the version info of the job worker

Returns:

the worker id for this worker

getWorkerInfoList

public List<WorkerInfo> getWorkerInfoList()

Returns:

a list of WorkerInfo objects representing the workers in Alluxio

setTaskPoolSize

public void setTaskPoolSize(int taskPoolSize)

Parameters:

taskPoolSize - the task pool size for the job workers

workerHeartbeat

public List<JobCommand> workerHeartbeat(JobWorkerHealth jobWorkerHealth,
                                        List<TaskInfo> taskInfoList)

Updates the tasks' status when a worker periodically heartbeats with the master, and sends the commands for the worker to execute.

Parameters:

jobWorkerHealth - the job worker health info

taskInfoList - the list of the task information

Returns:

the list of JobCommand to the worker

jobMasterHeartbeat

public JobMasterMetaCommand jobMasterHeartbeat(long masterId,
                                               JobMasterHeartbeatPOptions options)

Handles a heartbeat from a standby job master.

Parameters:

masterId - the job master id allocated by the primary job master

options - extra options

Returns:

the command to the standby job master

jobMasterRegister

public void jobMasterRegister(long masterId,
                              RegisterJobMasterPOptions options)
                       throws NotFoundException

Handles the register request from a standby job master.

Parameters:

masterId - the job master id

options - extra options

Throws:

NotFoundException

getMasterId

public long getMasterId(Address address)

Allocates an ID for the job master from the target address.

Parameters:

address - the address

Returns:

an allocated id