alluxio.master.block

Interface BlockMaster

All Superinterfaces:

Checkpointed, ContainerIdGenerable, Journaled, JournalEntryIterable, Master, Server<Boolean>

All Known Implementing Classes:

DefaultBlockMaster

public interface BlockMaster
extends Master, ContainerIdGenerable

Interface of the block master that manages the metadata for all the blocks and block workers in Alluxio.

Method Summary

Modifier and Type	Method and Description
void	commitBlock(long workerId, long usedBytesOnTier, String tierAlias, String mediumType, long blockId, long length) Marks a block as committed on a specific worker.
default void	commitBlockInUFS(long blockId, long length) Marks a block as committed, but without a worker location.
void	commitBlockInUFS(long blockId, long length, JournalContext context) Marks a block as committed, but without a worker location.
void	decommissionWorker(long workerId) Decommission a worker.
BlockInfo	getBlockInfo(long blockId)
List<BlockInfo>	getBlockInfoList(List<Long> blockIds) Retrieves information for the given list of block ids.
long	getBlockReplicaCount()
long	getCapacityBytes()
java.time.Clock	getClock() Returns the BlockMaster's clock so other components can align with the BlockMaster's time.
int	getDecommissionedWorkerCount()
StorageTierAssoc	getGlobalStorageTierAssoc()
long	getJournaledNextContainerId() Returns the internal JournaledNextContainerId.
int	getLostBlocksCount()
Iterator<Long>	getLostBlocksIterator() Returns an Iterator over the lost blocks.
int	getLostWorkerCount()
List<WorkerInfo>	getLostWorkersInfoList()
Map<String,Long>	getTotalBytesOnTiers()
long	getUniqueBlockCount()
long	getUsedBytes()
Map<String,Long>	getUsedBytesOnTiers()
MasterWorkerInfo	getWorker(long workerId) Returns the internal MasterWorkerInfo object to the caller.
Set<WorkerNetAddress>	getWorkerAddresses()
int	getWorkerCount()
long	getWorkerId(WorkerNetAddress workerNetAddress) Returns a worker id for the given worker, creating one if the worker is new.
List<WorkerInfo>	getWorkerInfoList()
List<WorkerLostStorageInfo>	getWorkerLostStorage()
List<WorkerInfo>	getWorkerReport(GetWorkerReportOptions options) Gets the worker information list for report CLI.
boolean	hasRegisterLease(long workerId) Verifies if the worker currently holds a RegisterLease.
boolean	isBlockLost(long blockId)
void	notifyWorkerId(long workerId, WorkerNetAddress workerNetAddress) Notify the worker id to a master.
void	registerLostWorkerFoundListener(java.util.function.Consumer<Address> function) Registers callback functions to use when lost workers become alive.
void	registerNewWorkerConfListener(java.util.function.BiConsumer<Address,List<ConfigProperty>> function) Registers callback functions to use when workers register with configuration.
void	registerWorkerDeleteListener(java.util.function.Consumer<Address> function) Registers callback functions to use when detecting lost workers.
void	registerWorkerLostListener(java.util.function.Consumer<Address> function) Registers callback functions to use when detecting lost workers.
void	releaseRegisterLease(long workerId) Releases the RegisterLease for the specified worker.
void	removeBlocks(Collection<Long> blockIds, boolean delete) Removes blocks from workers.
void	removeDecommissionedWorker(long workerId) Removes all associated metadata about the decommissioned worker from block master.
void	reportLostBlocks(List<Long> blockIds) Reports the ids of the blocks lost on workers.
Optional<RegisterLease>	tryAcquireRegisterLease(GetRegisterLeasePRequest request) Try to acquire a RegisterLease for the worker.
void	validateBlocks(java.util.function.Function<Long,Boolean> validator, boolean repair) Validates the integrity of blocks with respect to the validator.
Command	workerHeartbeat(long workerId, Map<String,Long> capacityBytesOnTiers, Map<String,Long> usedBytesOnTiers, List<Long> removedBlockIds, Map<Block.BlockLocation,List<Long>> addedBlocks, Map<String,StorageList> lostStorage, List<Metric> metrics) Updates metadata when a worker periodically heartbeats with the master.
void	workerRegister(long workerId, List<String> storageTiers, Map<String,Long> totalBytesOnTiers, Map<String,Long> usedBytesOnTiers, Map<Block.BlockLocation,List<Long>> currentBlocksOnLocation, Map<String,StorageList> lostStorage, RegisterWorkerPOptions options) Updates metadata when a worker registers with the master.
void	workerRegisterFinish(WorkerRegisterContext context) Completes the worker registration stream.
void	workerRegisterStream(WorkerRegisterContext context, RegisterWorkerPRequest chunk, boolean isFirstMsg) Handles messages in a register stream.

Methods inherited from interface alluxio.master.Master

createJournalContext, getMasterContext, getStandbyServices

Methods inherited from interface alluxio.master.journal.Journaled

applyAndJournal, processJournalEntry, resetState, restoreFromCheckpoint, writeToCheckpoint

Methods inherited from interface alluxio.master.journal.checkpoint.Checkpointed

getCheckpointName

Methods inherited from interface alluxio.master.journal.JournalEntryIterable

getJournalEntryIterator

Methods inherited from interface alluxio.Server

close, getDependencies, getName, getServices, start, stop

Methods inherited from interface alluxio.master.block.ContainerIdGenerable

getNewContainerId

Method Detail

getWorkerCount

int getWorkerCount()

Returns:

the number of live workers

getLostWorkerCount

int getLostWorkerCount()

Returns:

the number of lost workers

getDecommissionedWorkerCount

int getDecommissionedWorkerCount()

Returns:

the number of decommissioned workers

getCapacityBytes

long getCapacityBytes()

Returns:

the total capacity (in bytes) on all tiers, on all workers of Alluxio

getUniqueBlockCount

long getUniqueBlockCount()

Returns:

the unique block count on all workers of Alluxio

getBlockReplicaCount

long getBlockReplicaCount()

Returns:

the replica block count on all workers of Alluxio

getGlobalStorageTierAssoc

StorageTierAssoc getGlobalStorageTierAssoc()

Returns:

the global storage tier mapping

getUsedBytes

long getUsedBytes()

Returns:

the total used bytes on all tiers, on all workers of Alluxio

getWorkerInfoList

List<WorkerInfo> getWorkerInfoList()
                            throws UnavailableException

Returns:

a list of WorkerInfo objects representing the live workers in Alluxio

Throws:

UnavailableException

getLostWorkersInfoList

List<WorkerInfo> getLostWorkersInfoList()
                                 throws UnavailableException

Returns:

a list of WorkerInfos of lost workers

Throws:

UnavailableException

getWorkerAddresses

Set<WorkerNetAddress> getWorkerAddresses()
                                  throws UnavailableException

Returns:

a set of live worker addresses

Throws:

UnavailableException

getWorkerReport

List<WorkerInfo> getWorkerReport(GetWorkerReportOptions options)
                          throws UnavailableException,
                                 InvalidArgumentException

Gets the worker information list for report CLI.

Parameters:

options - the GetWorkerReportOptions defines the info range

Returns:

a list of WorkerInfo objects representing the workers in Alluxio

Throws:

UnavailableException

InvalidArgumentException

getWorkerLostStorage

List<WorkerLostStorageInfo> getWorkerLostStorage()

Returns:

a list of worker lost storage information

decommissionWorker

void decommissionWorker(long workerId)
                 throws Exception

Decommission a worker.

Parameters:

workerId - the WorkerInfo of worker to be decommissioned

Throws:

Exception

removeBlocks

void removeBlocks(Collection<Long> blockIds,
                  boolean delete)
           throws UnavailableException

Removes blocks from workers.

Parameters:

blockIds - a list of block ids to remove from Alluxio space

delete - whether to delete blocks' metadata in Master

Throws:

UnavailableException

validateBlocks

void validateBlocks(java.util.function.Function<Long,Boolean> validator,
                    boolean repair)
             throws UnavailableException

Validates the integrity of blocks with respect to the validator. A warning will be printed if blocks are invalid.

Parameters:

validator - a function returns true if the given block id is valid

repair - if true, deletes the invalid blocks

Throws:

UnavailableException - if the invalid blocks cannot be deleted

commitBlock

void commitBlock(long workerId,
                 long usedBytesOnTier,
                 String tierAlias,
                 String mediumType,
                 long blockId,
                 long length)
          throws NotFoundException,
                 UnavailableException

Marks a block as committed on a specific worker.

Parameters:

workerId - the worker id committing the block

usedBytesOnTier - the updated used bytes on the tier of the worker

tierAlias - the alias of the storage tier where the worker is committing the block to

mediumType - the medium type where the worker is committing the block to

blockId - the committing block id

length - the length of the block

Throws:

NotFoundException - if the workerId is not active

UnavailableException

commitBlockInUFS

default void commitBlockInUFS(long blockId,
                              long length)
                       throws UnavailableException

Marks a block as committed, but without a worker location. This means the block is only in ufs.

Parameters:

blockId - the id of the block to commit

length - the length of the block

Throws:

UnavailableException

commitBlockInUFS

void commitBlockInUFS(long blockId,
                      long length,
                      JournalContext context)

Marks a block as committed, but without a worker location. This means the block is only in ufs. Append any created journal entries to the included context.

Parameters:

blockId - the id of the block to commit

length - the length of the block

context - the journal context

getBlockInfo

BlockInfo getBlockInfo(long blockId)
                throws BlockInfoException,
                       UnavailableException

Parameters:

blockId - the block id to get information for

Returns:

the BlockInfo for the given block id

Throws:

BlockInfoException - if the block info is not found

UnavailableException

getBlockInfoList

List<BlockInfo> getBlockInfoList(List<Long> blockIds)
                          throws UnavailableException

Retrieves information for the given list of block ids.

Parameters:

blockIds - A list of block ids to retrieve the information for

Returns:

A list of BlockInfo objects corresponding to the input list of block ids. The list is in the same order as the input list

Throws:

UnavailableException

getTotalBytesOnTiers

Map<String,Long> getTotalBytesOnTiers()

Returns:

the total bytes on each storage tier

getUsedBytesOnTiers

Map<String,Long> getUsedBytesOnTiers()

Returns:

the used bytes on each storage tier

getWorkerId

long getWorkerId(WorkerNetAddress workerNetAddress)

Returns a worker id for the given worker, creating one if the worker is new.

Parameters:

workerNetAddress - the worker WorkerNetAddress

Returns:

the worker id for this worker

tryAcquireRegisterLease

Optional<RegisterLease> tryAcquireRegisterLease(GetRegisterLeasePRequest request)

Try to acquire a RegisterLease for the worker. If the lease is not granted, this will return empty immediately rather than blocking.

Parameters:

request - the request with all information for the master to make a decision with

Returns:

if empty, that means the lease is not granted

hasRegisterLease

boolean hasRegisterLease(long workerId)

Verifies if the worker currently holds a RegisterLease.

Parameters:

workerId - the worker ID

Returns:

whether a lease is found

releaseRegisterLease

void releaseRegisterLease(long workerId)

Releases the RegisterLease for the specified worker. If the worker currently does not hold a lease, return without throwing an error. The lease may have been recycled already.

Parameters:

workerId - the worker ID

workerRegister

void workerRegister(long workerId,
                    List<String> storageTiers,
                    Map<String,Long> totalBytesOnTiers,
                    Map<String,Long> usedBytesOnTiers,
                    Map<Block.BlockLocation,List<Long>> currentBlocksOnLocation,
                    Map<String,StorageList> lostStorage,
                    RegisterWorkerPOptions options)
             throws NotFoundException

Updates metadata when a worker registers with the master.

Parameters:

workerId - the worker id of the worker registering

storageTiers - a list of storage tier aliases in order of their position in the worker's hierarchy

totalBytesOnTiers - a mapping from storage tier alias to total bytes

usedBytesOnTiers - a mapping from storage tier alias to the used byes

currentBlocksOnLocation - a mapping from storage tier alias to a list of blocks

lostStorage - a mapping from storage tier alias to a list of lost storage paths

options - the options that may contain worker configuration

Throws:

NotFoundException - if workerId cannot be found

workerHeartbeat

Command workerHeartbeat(long workerId,
                        Map<String,Long> capacityBytesOnTiers,
                        Map<String,Long> usedBytesOnTiers,
                        List<Long> removedBlockIds,
                        Map<Block.BlockLocation,List<Long>> addedBlocks,
                        Map<String,StorageList> lostStorage,
                        List<Metric> metrics)

Updates metadata when a worker periodically heartbeats with the master.

Parameters:

workerId - the worker id

capacityBytesOnTiers - a mapping from tier alias to the capacity bytes

usedBytesOnTiers - a mapping from tier alias to the used bytes

removedBlockIds - a list of block ids removed from this worker

addedBlocks - a mapping from tier alias to the added blocks

lostStorage - a mapping from tier alias to lost storage paths

metrics - worker metrics

Returns:

an optional command for the worker to execute

isBlockLost

boolean isBlockLost(long blockId)

Parameters:

blockId - the block ID

Returns:

whether the block is considered lost in Alluxio

getLostBlocksIterator

Iterator<Long> getLostBlocksIterator()

Returns an Iterator over the lost blocks. Note that the iterator should not be shared across threads.

Returns:

an Iterator

getLostBlocksCount

int getLostBlocksCount()

Returns:

the number of lost blocks in Alluxio

reportLostBlocks

void reportLostBlocks(List<Long> blockIds)

Reports the ids of the blocks lost on workers.

Parameters:

blockIds - the ids of the lost blocks

registerLostWorkerFoundListener

void registerLostWorkerFoundListener(java.util.function.Consumer<Address> function)

Registers callback functions to use when lost workers become alive.

Parameters:

function - the function to register

registerWorkerLostListener

void registerWorkerLostListener(java.util.function.Consumer<Address> function)

Registers callback functions to use when detecting lost workers.

Parameters:

function - the function to register

registerWorkerDeleteListener

void registerWorkerDeleteListener(java.util.function.Consumer<Address> function)

Registers callback functions to use when detecting lost workers.

Parameters:

function - the function to register

registerNewWorkerConfListener

void registerNewWorkerConfListener(java.util.function.BiConsumer<Address,List<ConfigProperty>> function)

Registers callback functions to use when workers register with configuration.

Parameters:

function - the function to register

getWorker

MasterWorkerInfo getWorker(long workerId)
                    throws NotFoundException

Returns the internal MasterWorkerInfo object to the caller. This is specifically for the tests and the WorkerRegisterContext. Note that this operations on the object requires locking. See the javadoc of MasterWorkerInfo for how the locking should be carefully done. When in doubt, do not use this API. Find other methods in this class that exposes necessary information.

Parameters:

workerId - the worker ID

Returns:

the MasterWorkerInfo for the worker

Throws:

NotFoundException

workerRegisterStream

void workerRegisterStream(WorkerRegisterContext context,
                          RegisterWorkerPRequest chunk,
                          boolean isFirstMsg)

Handles messages in a register stream.

Parameters:

context - the stream context that contains the worker information

chunk - the message in a stream

isFirstMsg - whether the message is the 1st in a stream

workerRegisterFinish

void workerRegisterFinish(WorkerRegisterContext context)

Completes the worker registration stream.

Parameters:

context - the stream context to be closed

getClock

java.time.Clock getClock()

Returns the BlockMaster's clock so other components can align with the BlockMaster's time.

Returns:

the current clock

getJournaledNextContainerId

long getJournaledNextContainerId()

Returns the internal JournaledNextContainerId.

Returns:

JournaledNextContainerId

removeDecommissionedWorker

void removeDecommissionedWorker(long workerId)
                         throws NotFoundException

Removes all associated metadata about the decommissioned worker from block master. The worker to free must have been decommissioned.

Parameters:

workerId - the workerId of target worker

Throws:

NotFoundException

notifyWorkerId

void notifyWorkerId(long workerId,
                    WorkerNetAddress workerNetAddress)

Notify the worker id to a master.

Parameters:

workerId - the worker id

workerNetAddress - the worker address