alluxio.master.block

Class DefaultBlockMaster

java.lang.Object

alluxio.master.AbstractMaster

alluxio.master.CoreMaster

alluxio.master.block.DefaultBlockMaster

All Implemented Interfaces:

BlockMaster, ContainerIdGenerable, Checkpointed, Journaled, JournalEntryIterable, Master, Server<Boolean>

@NotThreadSafe
public class DefaultBlockMaster
extends CoreMaster
implements BlockMaster

This block master manages the metadata for all the blocks and block workers in Alluxio.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	DefaultBlockMaster.Metrics Class that contains metrics related to BlockMaster.

Field Summary

Fields inherited from class alluxio.master.CoreMaster

mBackupManager, mJournalSystem, mPort, mSafeModeManager, mStartTimeMs

Fields inherited from class alluxio.master.AbstractMaster

mClock, mMasterContext

Method Summary

Modifier and Type	Method and Description
void	close() Closes the server.
void	commitBlock(long workerId, long usedBytesOnTier, String tierAlias, String mediumType, long blockId, long length) Marks a block as committed on a specific worker.
void	commitBlockInUFS(long blockId, long length) Marks a block as committed, but without a worker location.
void	forgetAllWorkers() Forces all workers to be lost.
BlockInfo	getBlockInfo(long blockId)
List<BlockInfo>	getBlockInfoList(List<Long> blockIds) Retrieves information for the given list of block ids.
long	getCapacityBytes()
CheckpointName	getCheckpointName()
Set<Class<? extends Server>>	getDependencies()
StorageTierAssoc	getGlobalStorageTierAssoc()
CloseableIterator<Journal.JournalEntry>	getJournalEntryIterator()
int	getLostBlocksCount()
Iterator<Long>	getLostBlocksIterator() Returns an Iterator over the lost blocks.
int	getLostWorkerCount()
List<WorkerInfo>	getLostWorkersInfoList()
String	getName()
long	getNewContainerId()
Map<ServiceType,GrpcService>	getServices()
Map<String,Long>	getTotalBytesOnTiers()
long	getUsedBytes()
Map<String,Long>	getUsedBytesOnTiers()
Set<WorkerNetAddress>	getWorkerAddresses()
int	getWorkerCount()
long	getWorkerId(WorkerNetAddress workerNetAddress) Returns a worker id for the given worker, creating one if the worker is new.
List<WorkerInfo>	getWorkerInfoList()
List<WorkerLostStorageInfo>	getWorkerLostStorage()
List<WorkerInfo>	getWorkerReport(GetWorkerReportOptions options) Gets the worker information list for report CLI.
boolean	isBlockLost(long blockId)
boolean	processJournalEntry(Journal.JournalEntry entry) Attempts to apply a journal entry.
void	registerLostWorkerFoundListener(java.util.function.Consumer<Address> function) Registers callback functions to use when lost workers become alive.
void	registerNewWorkerConfListener(java.util.function.BiConsumer<Address,List<ConfigProperty>> function) Registers callback functions to use when workers register with configuration.
void	registerWorkerLostListener(java.util.function.Consumer<Address> function) Registers callback functions to use when detecting lost workers.
void	removeBlocks(List<Long> blockIds, boolean delete) Removes blocks from workers.
void	reportLostBlocks(List<Long> blockIds) Reports the ids of the blocks lost on workers.
void	resetState() Resets the object's journaled state.
void	start(Boolean isLeader) Starts the Alluxio server.
void	stop() Stops the Alluxio server.
void	validateBlocks(java.util.function.Function<Long,Boolean> validator, boolean repair) Validates the integrity of blocks with respect to the validator.
Command	workerHeartbeat(long workerId, Map<String,Long> capacityBytesOnTiers, Map<String,Long> usedBytesOnTiers, List<Long> removedBlockIds, Map<Block.BlockLocation,List<Long>> addedBlocks, Map<String,StorageList> lostStorage, List<Metric> metrics) Updates metadata when a worker periodically heartbeats with the master.
void	workerRegister(long workerId, List<String> storageTiers, Map<String,Long> totalBytesOnTiers, Map<String,Long> usedBytesOnTiers, Map<Block.BlockLocation,List<Long>> currentBlocksOnLocation, Map<String,StorageList> lostStorage, RegisterWorkerPOptions options) Updates metadata when a worker registers with the master.

Methods inherited from class alluxio.master.AbstractMaster

createJournalContext, getExecutorService, getMasterContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface alluxio.master.Master

createJournalContext, getMasterContext

Methods inherited from interface alluxio.master.journal.Journaled

applyAndJournal, restoreFromCheckpoint, writeToCheckpoint

Method Detail

getName

public String getName()

Specified by:

getName in interface Server<Boolean>

Returns:

the server's name

getServices

public Map<ServiceType,GrpcService> getServices()

Specified by:

getServices in interface Server<Boolean>

Returns:

a map from service names to gRPC serviced that serve RPCs for this server

processJournalEntry

public boolean processJournalEntry(Journal.JournalEntry entry)

Description copied from interface: Journaled

Attempts to apply a journal entry.

Specified by:

processJournalEntry in interface Journaled

Parameters:

entry - the entry to apply

Returns:

whether the entry type is supported by this journaled object

resetState

public void resetState()

Description copied from interface: Journaled

Resets the object's journaled state.

Specified by:

resetState in interface Journaled

getCheckpointName

public CheckpointName getCheckpointName()

Specified by:

getCheckpointName in interface Checkpointed

Returns:

a name for this checkpointed class

getJournalEntryIterator

public CloseableIterator<Journal.JournalEntry> getJournalEntryIterator()

Specified by:

getJournalEntryIterator in interface JournalEntryIterable

Returns:

an CloseableIterator that iterates all the journal entries

start

public void start(Boolean isLeader)
           throws IOException

Description copied from interface: Server

Starts the Alluxio server.

Specified by:

start in interface Server<Boolean>

Overrides:

start in class AbstractMaster

Parameters:

isLeader - the start options

Throws:

IOException

stop

public void stop()
          throws IOException

Description copied from interface: Server

Stops the Alluxio server. Here, anything created or started in #start(T) should be cleaned up and shutdown.

Specified by:

stop in interface Server<Boolean>

Overrides:

stop in class AbstractMaster

Throws:

IOException

close

public void close()
           throws IOException

Description copied from interface: Server

Closes the server.

Specified by:

close in interface Server<Boolean>

Overrides:

close in class AbstractMaster

Throws:

IOException

getWorkerCount

public int getWorkerCount()

Specified by:

getWorkerCount in interface BlockMaster

Returns:

the number of live workers

getLostWorkerCount

public int getLostWorkerCount()

Specified by:

getLostWorkerCount in interface BlockMaster

Returns:

the number of lost workers

getCapacityBytes

public long getCapacityBytes()

Specified by:

getCapacityBytes in interface BlockMaster

Returns:

the total capacity (in bytes) on all tiers, on all workers of Alluxio

getGlobalStorageTierAssoc

public StorageTierAssoc getGlobalStorageTierAssoc()

Specified by:

getGlobalStorageTierAssoc in interface BlockMaster

Returns:

the global storage tier mapping

getUsedBytes

public long getUsedBytes()

Specified by:

getUsedBytes in interface BlockMaster

Returns:

the total used bytes on all tiers, on all workers of Alluxio

getWorkerInfoList

public List<WorkerInfo> getWorkerInfoList()
                                   throws UnavailableException

Specified by:

getWorkerInfoList in interface BlockMaster

Returns:

a list of WorkerInfo objects representing the live workers in Alluxio

Throws:

UnavailableException

getLostWorkersInfoList

public List<WorkerInfo> getLostWorkersInfoList()
                                        throws UnavailableException

Specified by:

getLostWorkersInfoList in interface BlockMaster

Returns:

a list of WorkerInfos of lost workers

Throws:

UnavailableException

getWorkerAddresses

public Set<WorkerNetAddress> getWorkerAddresses()
                                         throws UnavailableException

Specified by:

getWorkerAddresses in interface BlockMaster

Returns:

a set of live worker addresses

Throws:

UnavailableException

getWorkerReport

public List<WorkerInfo> getWorkerReport(GetWorkerReportOptions options)
                                 throws UnavailableException,
                                        InvalidArgumentException

Description copied from interface: BlockMaster

Gets the worker information list for report CLI.

Specified by:

getWorkerReport in interface BlockMaster

Parameters:

options - the GetWorkerReportOptions defines the info range

Returns:

a list of WorkerInfo objects representing the workers in Alluxio

Throws:

UnavailableException

InvalidArgumentException

getWorkerLostStorage

public List<WorkerLostStorageInfo> getWorkerLostStorage()

Specified by:

getWorkerLostStorage in interface BlockMaster

Returns:

a list of worker lost storage information

removeBlocks

public void removeBlocks(List<Long> blockIds,
                         boolean delete)
                  throws UnavailableException

Description copied from interface: BlockMaster

Removes blocks from workers.

Specified by:

removeBlocks in interface BlockMaster

Parameters:

blockIds - a list of block ids to remove from Alluxio space

delete - whether to delete blocks' metadata in Master

Throws:

UnavailableException

validateBlocks

public void validateBlocks(java.util.function.Function<Long,Boolean> validator,
                           boolean repair)
                    throws UnavailableException

Description copied from interface: BlockMaster

Validates the integrity of blocks with respect to the validator. A warning will be printed if blocks are invalid.

Specified by:

validateBlocks in interface BlockMaster

Parameters:

validator - a function returns true if the given block id is valid

repair - if true, deletes the invalid blocks

Throws:

UnavailableException - if the invalid blocks cannot be deleted

getNewContainerId

public long getNewContainerId()
                       throws UnavailableException

Specified by:

getNewContainerId in interface ContainerIdGenerable

Returns:

a new block container id

Throws:

UnavailableException

commitBlock

public void commitBlock(long workerId,
                        long usedBytesOnTier,
                        String tierAlias,
                        String mediumType,
                        long blockId,
                        long length)
                 throws NotFoundException,
                        UnavailableException

Description copied from interface: BlockMaster

Marks a block as committed on a specific worker.

Specified by:

commitBlock in interface BlockMaster

Parameters:

workerId - the worker id committing the block

usedBytesOnTier - the updated used bytes on the tier of the worker

tierAlias - the alias of the storage tier where the worker is committing the block to

mediumType - the medium type where the worker is committing the block to

blockId - the committing block id

length - the length of the block

Throws:

NotFoundException - if the workerId is not active

UnavailableException

commitBlockInUFS

public void commitBlockInUFS(long blockId,
                             long length)
                      throws UnavailableException

Description copied from interface: BlockMaster

Marks a block as committed, but without a worker location. This means the block is only in ufs.

Specified by:

commitBlockInUFS in interface BlockMaster

Parameters:

blockId - the id of the block to commit

length - the length of the block

Throws:

UnavailableException

getBlockInfo

public BlockInfo getBlockInfo(long blockId)
                       throws BlockInfoException,
                              UnavailableException

Specified by:

getBlockInfo in interface BlockMaster

Parameters:

blockId - the block id to get information for

Returns:

the BlockInfo for the given block id

Throws:

BlockInfoException - if the block info is not found

UnavailableException

getBlockInfoList

public List<BlockInfo> getBlockInfoList(List<Long> blockIds)
                                 throws UnavailableException

Description copied from interface: BlockMaster

Retrieves information for the given list of block ids.

Specified by:

getBlockInfoList in interface BlockMaster

Parameters:

blockIds - A list of block ids to retrieve the information for

Returns:

A list of BlockInfo objects corresponding to the input list of block ids. The list is in the same order as the input list

Throws:

UnavailableException

getTotalBytesOnTiers

public Map<String,Long> getTotalBytesOnTiers()

Specified by:

getTotalBytesOnTiers in interface BlockMaster

Returns:

the total bytes on each storage tier

getUsedBytesOnTiers

public Map<String,Long> getUsedBytesOnTiers()

Specified by:

getUsedBytesOnTiers in interface BlockMaster

Returns:

the used bytes on each storage tier

getWorkerId

public long getWorkerId(WorkerNetAddress workerNetAddress)

Description copied from interface: BlockMaster

Returns a worker id for the given worker, creating one if the worker is new.

Specified by:

getWorkerId in interface BlockMaster

Parameters:

workerNetAddress - the worker WorkerNetAddress

Returns:

the worker id for this worker

workerRegister

public void workerRegister(long workerId,
                           List<String> storageTiers,
                           Map<String,Long> totalBytesOnTiers,
                           Map<String,Long> usedBytesOnTiers,
                           Map<Block.BlockLocation,List<Long>> currentBlocksOnLocation,
                           Map<String,StorageList> lostStorage,
                           RegisterWorkerPOptions options)
                    throws NotFoundException

Description copied from interface: BlockMaster

Updates metadata when a worker registers with the master.

Specified by:

workerRegister in interface BlockMaster

Parameters:

workerId - the worker id of the worker registering

storageTiers - a list of storage tier aliases in order of their position in the worker's hierarchy

totalBytesOnTiers - a mapping from storage tier alias to total bytes

usedBytesOnTiers - a mapping from storage tier alias to the used byes

currentBlocksOnLocation - a mapping from storage tier alias to a list of blocks

lostStorage - a mapping from storage tier alias to a list of lost storage paths

options - the options that may contain worker configuration

Throws:

NotFoundException - if workerId cannot be found

workerHeartbeat

public Command workerHeartbeat(long workerId,
                               Map<String,Long> capacityBytesOnTiers,
                               Map<String,Long> usedBytesOnTiers,
                               List<Long> removedBlockIds,
                               Map<Block.BlockLocation,List<Long>> addedBlocks,
                               Map<String,StorageList> lostStorage,
                               List<Metric> metrics)

Description copied from interface: BlockMaster

Updates metadata when a worker periodically heartbeats with the master.

Specified by:

workerHeartbeat in interface BlockMaster

Parameters:

workerId - the worker id

capacityBytesOnTiers - a mapping from tier alias to the capacity bytes

usedBytesOnTiers - a mapping from tier alias to the used bytes

removedBlockIds - a list of block ids removed from this worker

addedBlocks - a mapping from tier alias to the added blocks

lostStorage - a mapping from tier alias to lost storage paths

metrics - worker metrics

Returns:

an optional command for the worker to execute

isBlockLost

public boolean isBlockLost(long blockId)

Specified by:

isBlockLost in interface BlockMaster

Parameters:

blockId - the block ID

Returns:

whether the block is considered lost in Alluxio

getLostBlocksIterator

public Iterator<Long> getLostBlocksIterator()

Description copied from interface: BlockMaster

Returns an Iterator over the lost blocks. Note that the iterator should not be shared across threads.

Specified by:

getLostBlocksIterator in interface BlockMaster

Returns:

an Iterator

getLostBlocksCount

public int getLostBlocksCount()

Specified by:

getLostBlocksCount in interface BlockMaster

Returns:

the number of lost blocks in Alluxio

reportLostBlocks

public void reportLostBlocks(List<Long> blockIds)

Description copied from interface: BlockMaster

Reports the ids of the blocks lost on workers.

Specified by:

reportLostBlocks in interface BlockMaster

Parameters:

blockIds - the ids of the lost blocks

getDependencies

public Set<Class<? extends Server>> getDependencies()

Specified by:

getDependencies in interface Server<Boolean>

Overrides:

getDependencies in class AbstractMaster

Returns:

the server dependencies

forgetAllWorkers

public void forgetAllWorkers()

Forces all workers to be lost. This should only be used for testing.

registerLostWorkerFoundListener

public void registerLostWorkerFoundListener(java.util.function.Consumer<Address> function)

Description copied from interface: BlockMaster

Registers callback functions to use when lost workers become alive.

Specified by:

registerLostWorkerFoundListener in interface BlockMaster

Parameters:

function - the function to register

registerWorkerLostListener

public void registerWorkerLostListener(java.util.function.Consumer<Address> function)

Description copied from interface: BlockMaster

Registers callback functions to use when detecting lost workers.

Specified by:

registerWorkerLostListener in interface BlockMaster

Parameters:

function - the function to register

registerNewWorkerConfListener

public void registerNewWorkerConfListener(java.util.function.BiConsumer<Address,List<ConfigProperty>> function)

Description copied from interface: BlockMaster

Registers callback functions to use when workers register with configuration.

Specified by:

registerNewWorkerConfListener in interface BlockMaster

Parameters:

function - the function to register