alluxio.master.block.meta

Class MasterWorkerInfo

java.lang.Object

alluxio.master.block.meta.MasterWorkerInfo

@NotThreadSafe
public final class MasterWorkerInfo
extends Object

This class embeds all metadata for an Alluxio worker. This class is not thread safe, so external locking is required. There are multiple locks in this object, each guarding a group of metadata. The metadata fields are separated into a few different groups. Each group has its corresponding locking mechanism. The MasterWorkerInfo has the following groups of metadata: 1. Metadata like ID, address etc., represented by a StaticWorkerMeta object. This group is thread safe, meaning no locking is required. 2. Worker last updated timestamp. This is thread safe, meaning no locking is required. 3. Worker register status. This is guarded by a StampedLock.asReadWriteLock(). 4. Worker resource usage, represented by a WorkerUsageMeta object. This is guarded by a StampedLock.asReadWriteLock(). 5. Worker block lists, including the present blocks and blocks to be removed from the worker. This is guarded by a StampedLock.asReadWriteLock(). When accessing certain fields in this object, external locking is required. As listed above, group 1 and 2 are thread safe and do not require external locking. Group 3, 4, and 5 require external locking. Locking can be done with lockWorkerMeta(EnumSet, boolean). This method returns a LockResource which can be managed by try-finally. Internally, WorkerMetaLock is used to manage the corresponding internal locks with an order and unlocking them properly on close. If the fields are only read, shared locks should be acquired and released as below:

   try (LockResource r = lockWorkerMeta(
       EnumSet.of(WorkerMetaLockSection.USAGE), true)) {
     ...
   }
 

If the fields are updated, exclusive locks should be acquired and released as below:

   try (LockResource r = lockWorkerMeta(
       EnumSet.of(
           WorkerMetaLockSection.STATUS,
           WorkerMetaLockSection.USAGE,
           WorkerMetaLockSection.BLOCKS)), true)) {
     ...
   }
 

The locks are internally StampedLock which are NOT reentrant! We chose StampedLock instead of ReentrantReadWriteLock because the latter does not allow the write lock to be grabbed in one thread but later released in another thread. This is undesirable because when the worker registers in a stream, the thread that handles the 1st message will acquire the lock and the thread that handles the complete signal will be responsible for releasing the lock. In the current gRPC architecture it is impossible to enforce the two threads to be the same. Because then locks are not reentrant, you must be extra careful NOT to acquire the lock while holding, because that will result in a deadlock! This is especially the case for write locks. The current write lock holders include the following: 1. In DefaultBlockMaster, the methods related to worker register/heartbeat, and block removal/commit. 2. In WorkerRegisterContext, to write locks are held throughout the lifecycle. 3. In DefaultBlockMaster.LostWorkerDetectionHeartbeatExecutor#heartbeat(long)

Field Summary

Fields

Modifier and Type	Field and Description
boolean	mIsRegistered If true, the worker is considered registered.

Constructor Summary

Constructors

Constructor and Description
MasterWorkerInfo(long id, WorkerNetAddress address) Creates a new instance of MasterWorkerInfo.

Method Summary

Modifier and Type	Method and Description
void	addBlock(long blockId) Adds a block to the worker.
void	addLostStorage(Map<String,StorageList> lostStorage) Adds new worker lost storage paths.
WorkerInfo	generateWorkerInfo(Set<GetWorkerReportOptions.WorkerInfoField> fieldRange, WorkerState workerState) Gets the selected field information for this worker.
long	getAvailableBytes() You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.USAGE specified.
int	getBlockCount() Return the block count of this worker.
Set<Long>	getBlocks() You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.BLOCKS specified.
BuildVersion	getBuildVersion() Get the build version of the worker.
long	getCapacityBytes() You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.USAGE specified.
Map<String,Long>	getFreeBytesOnTiers() You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.USAGE specified.
long	getId() No locking required.
long	getLastUpdatedTimeMs() No locking required.
Map<String,List<String>>	getLostStorage() You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.USAGE specified.
long	getStartTime() No locking required.
int	getToRemoveBlockCount() Returns the number of blocks that should be removed from the worker.
Set<Long>	getToRemoveBlocks() You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.BLOCKS specified.
Map<String,Long>	getTotalBytesOnTiers() You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.USAGE specified.
long	getUsedBytes()
Map<String,Long>	getUsedBytesOnTiers() You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.USAGE specified.
WorkerNetAddress	getWorkerAddress() StaticWorkerMeta is thread safe so the value can be read without locking.
boolean	hasLostStorage() You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.USAGE specified.
boolean	isRegistered() You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.STATUS specified.
LockResource	lockWorkerMeta(EnumSet<WorkerMetaLockSection> lockTypes, boolean isShared) Locks the corresponding locks on the metadata groups.
LockResource	lockWorkerMetaForInfo(Set<GetWorkerReportOptions.WorkerInfoField> fieldRange) Finds the read locks necessary for required worker information.
void	markAllBlocksToRemove() Marks all the blocks on the worker to be removed.
Set<Long>	register(StorageTierAssoc globalStorageTierAssoc, List<String> storageTierAliases, Map<String,Long> totalBytesOnTiers, Map<String,Long> usedBytesOnTiers, Set<Long> blocks) Marks the worker as registered, while updating all of its metadata.
void	removeBlockFromWorkerMeta(long blockId) Removes a block from the worker.
void	scheduleRemoveFromWorker(long blockId) Remove the block from the worker metadata and add to the to-remove list.
void	setBuildVersion(BuildVersion buildVersion) Sets the build version of the worker.
String	toString()
void	updateCapacityBytes(Map<String,Long> capacityBytesOnTiers) Sets the capacity of the worker in bytes.
void	updateLastUpdatedTimeMs() Updates the last updated time of the worker in ms.
void	updateToRemovedBlock(boolean add, long blockId) Adds or removes a block from the to-be-removed blocks set of the worker.
void	updateUsage(StorageTierAssoc globalStorageTierAssoc, List<String> storageTiers, Map<String,Long> totalBytesOnTiers, Map<String,Long> usedBytesOnTiers) Updates the worker storage usage.
void	updateUsedBytes(Map<String,Long> usedBytesOnTiers) Sets the used space of the worker in bytes.
void	updateUsedBytes(String tierAlias, long usedBytesOnTier) Sets the used space of the worker in bytes.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

mIsRegistered

public boolean mIsRegistered

If true, the worker is considered registered.

Constructor Detail

MasterWorkerInfo

public MasterWorkerInfo(long id,
                        WorkerNetAddress address)

Creates a new instance of MasterWorkerInfo.

Parameters:

id - the worker id to use

address - the worker address to use

Method Detail

register

public Set<Long> register(StorageTierAssoc globalStorageTierAssoc,
                          List<String> storageTierAliases,
                          Map<String,Long> totalBytesOnTiers,
                          Map<String,Long> usedBytesOnTiers,
                          Set<Long> blocks)

Marks the worker as registered, while updating all of its metadata. Write locks on mStatusLock, mUsageLock and mBlockListLock are required. You should lock externally with lockWorkerMeta(EnumSet, boolean) with all three lock types specified:

   try (LockResource r = worker.lockWorkerMeta(EnumSet.of(
       WorkerMetaLockSection.STATUS,
       WorkerMetaLockSection.USAGE,
       WorkerMetaLockSection.BLOCKS), false)) {
     register(...);
   }
 

Parameters:

globalStorageTierAssoc - global mapping between storage aliases and ordinal position

storageTierAliases - list of storage tier aliases in order of their position in the hierarchy

totalBytesOnTiers - mapping from storage tier alias to total bytes

usedBytesOnTiers - mapping from storage tier alias to used byes

blocks - set of block ids on this worker

Returns:

A Set of blocks removed (or lost) from this worker

addBlock

public void addBlock(long blockId)

Adds a block to the worker. You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.BLOCKS specified. An exclusive lock is required.

Parameters:

blockId - the id of the block to be added

removeBlockFromWorkerMeta

public void removeBlockFromWorkerMeta(long blockId)

Removes a block from the worker. This is typically called when we know the block has been removed from the worker. You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.BLOCKS specified. An exclusive lock is required.

Parameters:

blockId - the id of the block to be removed

scheduleRemoveFromWorker

public void scheduleRemoveFromWorker(long blockId)

Remove the block from the worker metadata and add to the to-remove list. The next worker heartbeat will issue the remove command to the worker so the block is deleted later. You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.BLOCKS specified. An exclusive lock is required.

Parameters:

blockId - the block ID

addLostStorage

public void addLostStorage(Map<String,StorageList> lostStorage)

Adds new worker lost storage paths. You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.USAGE specified. An exclusive lock is required.

Parameters:

lostStorage - the lost storage to add

generateWorkerInfo

public WorkerInfo generateWorkerInfo(Set<GetWorkerReportOptions.WorkerInfoField> fieldRange,
                                     WorkerState workerState)

Gets the selected field information for this worker. You should lock externally with lockWorkerMetaForInfo(Set). The required locks will be determined internally based on the fields.

Parameters:

fieldRange - the client selected fields

workerState - the worker state

Returns:

generated worker information

getWorkerAddress

public WorkerNetAddress getWorkerAddress()

StaticWorkerMeta is thread safe so the value can be read without locking.

Returns:

the worker's address

getAvailableBytes

public long getAvailableBytes()

You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.USAGE specified. A shared lock is required.

Returns:

the available space of the worker in bytes

getBlocks

public Set<Long> getBlocks()

You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.BLOCKS specified. A shared lock is required. This returns a copy so the lock can be released when this method returns.

Returns:

ids of all blocks the worker contains

getBlockCount

public int getBlockCount()

Return the block count of this worker.

Returns:

the block count of this worker

getCapacityBytes

public long getCapacityBytes()

You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.USAGE specified. A shared lock is required.

Returns:

the capacity of the worker in bytes

getId

public long getId()

No locking required.

Returns:

the id of the worker

getLastUpdatedTimeMs

public long getLastUpdatedTimeMs()

No locking required.

Returns:

the last updated time of the worker in ms

getToRemoveBlocks

public Set<Long> getToRemoveBlocks()

You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.BLOCKS specified. A shared lock is required. This returns a copy so the lock can be released after this method returns.

Returns:

ids of blocks the worker should remove

getUsedBytes

public long getUsedBytes()

Returns:

used space of the worker in bytes

getTotalBytesOnTiers

public Map<String,Long> getTotalBytesOnTiers()

You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.USAGE specified. A shared lock is required.

Returns:

the total bytes on each storage tier

getUsedBytesOnTiers

public Map<String,Long> getUsedBytesOnTiers()

You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.USAGE specified. A shared lock is required.

Returns:

the used bytes on each storage tier

getStartTime

public long getStartTime()

No locking required.

Returns:

the start time in milliseconds

isRegistered

public boolean isRegistered()

You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.STATUS specified. A shared lock is required.

Returns:

whether the worker has been registered yet

getFreeBytesOnTiers

public Map<String,Long> getFreeBytesOnTiers()

You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.USAGE specified. A shared lock is required.

Returns:

the free bytes on each storage tier

getLostStorage

public Map<String,List<String>> getLostStorage()

You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.USAGE specified. A shared lock is required. This returns a copy so the lock can be released after the map is returned.

Returns:

the map from tier alias to lost storage paths in this worker

hasLostStorage

public boolean hasLostStorage()

You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.USAGE specified. A shared lock is required.

Returns:

true if this worker has lost storage, false otherwise

toString

public String toString()

Overrides:

toString in class Object

updateLastUpdatedTimeMs

public void updateLastUpdatedTimeMs()

Updates the last updated time of the worker in ms. No locking is required.

updateToRemovedBlock

public void updateToRemovedBlock(boolean add,
                                 long blockId)

Adds or removes a block from the to-be-removed blocks set of the worker. You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.BLOCKS specified. An exclusive lock is required.

Parameters:

add - true if to add, to remove otherwise

blockId - the id of the block to be added or removed

updateCapacityBytes

public void updateCapacityBytes(Map<String,Long> capacityBytesOnTiers)

Sets the capacity of the worker in bytes. You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.USAGE specified. An exclusive lock is required.

Parameters:

capacityBytesOnTiers - used bytes on each storage tier

updateUsedBytes

public void updateUsedBytes(Map<String,Long> usedBytesOnTiers)

Sets the used space of the worker in bytes. You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.USAGE specified. An exclusive lock is required.

Parameters:

usedBytesOnTiers - used bytes on each storage tier

updateUsedBytes

public void updateUsedBytes(String tierAlias,
                            long usedBytesOnTier)

Sets the used space of the worker in bytes. You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.USAGE specified. An exclusive lock is required.

Parameters:

tierAlias - alias of storage tier

usedBytesOnTier - used bytes on certain storage tier

lockWorkerMeta

public LockResource lockWorkerMeta(EnumSet<WorkerMetaLockSection> lockTypes,
                                   boolean isShared)

Locks the corresponding locks on the metadata groups. See the javadoc for this class for example usage. The locks will be acquired in order and later released in the opposite order. The locks can either be shared or exclusive. The isShared flag will apply to all the locks acquired here. This returns a LockResource which can be managed by a try-finally block. See javadoc for WorkerMetaLock for more details about the internals.

Parameters:

lockTypes - the locks

isShared - if false, the locking is exclusive

Returns:

a LockResource of the WorkerMetaLock

getToRemoveBlockCount

public int getToRemoveBlockCount()

Returns the number of blocks that should be removed from the worker.

Returns:

the count

updateUsage

public void updateUsage(StorageTierAssoc globalStorageTierAssoc,
                        List<String> storageTiers,
                        Map<String,Long> totalBytesOnTiers,
                        Map<String,Long> usedBytesOnTiers)

Updates the worker storage usage. You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.USAGE specified. An exclusive lock is required.

Parameters:

globalStorageTierAssoc - storage tier setup from configuration

storageTiers - the storage tiers

totalBytesOnTiers - the capacity of each tier

usedBytesOnTiers - the current usage of each tier

markAllBlocksToRemove

public void markAllBlocksToRemove()

Marks all the blocks on the worker to be removed. This is called at the beginning of a register stream, where we do not know what blocks are no longer on the worker in mBlocks. First all blocks will be marked to-be-removed, then in the stream when we see a block list, those blocks will be added to mBlocks and removed from mToRemoveBlocks. In this way, at the end of the stream, mToRemoveBlocks contains only the blocks that no longer exist on the worker. You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.BLOCKS specified. An exclusive lock is required.

lockWorkerMetaForInfo

public LockResource lockWorkerMetaForInfo(Set<GetWorkerReportOptions.WorkerInfoField> fieldRange)

Finds the read locks necessary for required worker information. Locks the corresponding read locks for the specified worker information fields. This is a wrapper of lockWorkerMeta(EnumSet, boolean)

Parameters:

fieldRange - a set of GetWorkerReportOptions.WorkerInfoField

Returns:

a LockResource of the WorkerMetaLock

setBuildVersion

public void setBuildVersion(BuildVersion buildVersion)

Sets the build version of the worker. BuildVersion is reported by the worker in the register request.

Parameters:

buildVersion - the BuildVersion of the worker

getBuildVersion

public BuildVersion getBuildVersion()

Get the build version of the worker. This is used to monitor cluster status when performing rolling upgrades.

Returns:

the BuildVersion of the worker