alluxio.master.block.meta

Class MasterWorkerInfo

java.lang.Object

alluxio.master.block.meta.MasterWorkerInfo

@NotThreadSafe
public final class MasterWorkerInfo
extends Object

This class embeds all metadata for an Alluxio worker. This class is not thread safe, so external locking is required. There are multiple locks in this object, each guarding a group of metadata. The metadata fields are separated into a few different groups. Each group has its corresponding locking mechanism. The MasterWorkerInfo has the following groups of metadata: 1. Metadata like ID, address etc, represented by a StaticWorkerMeta object. This group is thread safe, meaning no locking is required. 2. Worker last updated timestamp. This is thread safe, meaning no locking is required. 3. Worker register status. This is guarded by a ReentrantReadWriteLock. 4. Worker resource usage, represented by a WorkerUsageMeta object. This is guarded by a ReentrantReadWriteLock. 5. Worker block lists, including the present blocks and blocks to be removed from the worker. This is guarded by a ReentrantReadWriteLock. When accessing certain fields in this object, external locking is required. As listed above, group 1 and 2 are thread safe and do not require external locking. Group 3, 4, and 5 require external locking. Locking can be done with lockWorkerMeta(EnumSet, boolean). This method returns a LockResource which can be managed by try-finally. Internally, WorkerMetaLock is used to manage the corresponding internal locks with an order and unlocking them properly on close. If the fields are only read, shared locks should be acquired and released as below:

   try (LockResource r = lockWorkerMeta(
       EnumSet.of(WorkerMetaLockSection.USAGE), true)) {
     ...
   }
 

If the fields are updated, exclusive locks should be acquired and released as below:

   try (LockResource r = lockWorkerMeta(
       EnumSet.of(
           WorkerMetaLockSection.STATUS,
           WorkerMetaLockSection.USAGE,
           WorkerMetaLockSection.BLOCKS)), true)) {
     ...
   }
 

Field Summary

Fields

Modifier and Type	Field and Description
boolean	mIsRegistered If true, the worker is considered registered.

Constructor Summary

Constructors

Constructor and Description
MasterWorkerInfo(long id, WorkerNetAddress address) Creates a new instance of MasterWorkerInfo.

Method Summary

Modifier and Type	Method and Description
void	addBlock(long blockId) Adds a block to the worker.
void	addLostStorage(Map<String,StorageList> lostStorage) Adds new worker lost storage paths.
void	addLostStorage(String tierAlias, String dirPath) Adds a new worker lost storage path.
WorkerInfo	generateWorkerInfo(Set<GetWorkerReportOptions.WorkerInfoField> fieldRange, boolean isLiveWorker) Gets the selected field information for this worker.
long	getAvailableBytes() You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.USAGE specified.
long	getBlockCount() Return the block count of this worker.
Set<Long>	getBlocks() You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.BLOCKS specified.
long	getCapacityBytes() You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.USAGE specified.
Map<String,Long>	getFreeBytesOnTiers() You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.USAGE specified.
long	getId() No locking required.
long	getLastUpdatedTimeMs() No locking required.
Map<String,List<String>>	getLostStorage() You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.USAGE specified.
long	getStartTime() No locking required.
StorageTierAssoc	getStorageTierAssoc() You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.USAGE specified.
List<Long>	getToRemoveBlocks() You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.BLOCKS specified.
Map<String,Long>	getTotalBytesOnTiers() You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.USAGE specified.
long	getUsedBytes()
Map<String,Long>	getUsedBytesOnTiers() You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.USAGE specified.
WorkerNetAddress	getWorkerAddress() StaticWorkerMeta is thread safe so the value can be read without locking.
boolean	hasLostStorage() You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.USAGE specified.
boolean	isRegistered() You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.STATUS specified.
LockResource	lockWorkerMeta(EnumSet<WorkerMetaLockSection> lockTypes, boolean isShared) Locks the corresponding locks on the metadata groups.
Set<Long>	register(StorageTierAssoc globalStorageTierAssoc, List<String> storageTierAliases, Map<String,Long> totalBytesOnTiers, Map<String,Long> usedBytesOnTiers, Set<Long> blocks) Marks the worker as registered, while updating all of its metadata.
void	removeBlock(long blockId) Removes a block from the worker.
String	toString()
void	updateCapacityBytes(Map<String,Long> capacityBytesOnTiers) Sets the capacity of the worker in bytes.
void	updateLastUpdatedTimeMs() Updates the last updated time of the worker in ms.
void	updateToRemovedBlock(boolean add, long blockId) Adds or removes a block from the to-be-removed blocks set of the worker.
void	updateUsedBytes(Map<String,Long> usedBytesOnTiers) Sets the used space of the worker in bytes.
void	updateUsedBytes(String tierAlias, long usedBytesOnTier) Sets the used space of the worker in bytes.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

mIsRegistered

public boolean mIsRegistered

If true, the worker is considered registered.

Constructor Detail

MasterWorkerInfo

public MasterWorkerInfo(long id,
                        WorkerNetAddress address)

Creates a new instance of MasterWorkerInfo.

Parameters:

id - the worker id to use

address - the worker address to use

Method Detail

register

public Set<Long> register(StorageTierAssoc globalStorageTierAssoc,
                          List<String> storageTierAliases,
                          Map<String,Long> totalBytesOnTiers,
                          Map<String,Long> usedBytesOnTiers,
                          Set<Long> blocks)

Marks the worker as registered, while updating all of its metadata. Write locks on mStatusLock, mUsageLock and mBlockListLock are required. You should lock externally with lockWorkerMeta(EnumSet, boolean) with all three lock types specified:

   try (LockResource r = worker.lockWorkerMeta(EnumSet.of(
       WorkerMetaLockSection.STATUS,
       WorkerMetaLockSection.USAGE,
       WorkerMetaLockSection.BLOCKS), false)) {
     register(...);
   }
 

Parameters:

globalStorageTierAssoc - global mapping between storage aliases and ordinal position

storageTierAliases - list of storage tier aliases in order of their position in the hierarchy

totalBytesOnTiers - mapping from storage tier alias to total bytes

usedBytesOnTiers - mapping from storage tier alias to used byes

blocks - set of block ids on this worker

Returns:

A Set of blocks removed (or lost) from this worker

addBlock

public void addBlock(long blockId)

Adds a block to the worker. You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.BLOCKS specified. An exclusive lock is required.

Parameters:

blockId - the id of the block to be added

removeBlock

public void removeBlock(long blockId)

Removes a block from the worker. You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.BLOCKS specified. An exclusive lock is required.

Parameters:

blockId - the id of the block to be removed

addLostStorage

public void addLostStorage(String tierAlias,
                           String dirPath)

Adds a new worker lost storage path. You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.USAGE specified. An exclusive lock is required.

Parameters:

tierAlias - the tier alias

dirPath - the lost storage path

addLostStorage

public void addLostStorage(Map<String,StorageList> lostStorage)

Adds new worker lost storage paths. You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.USAGE specified. An exclusive lock is required.

Parameters:

lostStorage - the lost storage to add

generateWorkerInfo

public WorkerInfo generateWorkerInfo(Set<GetWorkerReportOptions.WorkerInfoField> fieldRange,
                                     boolean isLiveWorker)

Gets the selected field information for this worker. You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.USAGE specified. A shared lock is required.

Parameters:

fieldRange - the client selected fields

isLiveWorker - the worker is live or not

Returns:

generated worker information

getWorkerAddress

public WorkerNetAddress getWorkerAddress()

StaticWorkerMeta is thread safe so the value can be read without locking.

Returns:

the worker's address

getAvailableBytes

public long getAvailableBytes()

You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.USAGE specified. A shared lock is required.

Returns:

the available space of the worker in bytes

getBlocks

public Set<Long> getBlocks()

You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.BLOCKS specified. A shared lock is required. This returns a copy so the lock can be released when this method returns.

Returns:

ids of all blocks the worker contains

getBlockCount

public long getBlockCount()

Return the block count of this worker.

Returns:

the block count of this worker

getCapacityBytes

public long getCapacityBytes()

You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.USAGE specified. A shared lock is required.

Returns:

the capacity of the worker in bytes

getId

public long getId()

No locking required.

Returns:

the id of the worker

getLastUpdatedTimeMs

public long getLastUpdatedTimeMs()

No locking required.

Returns:

the last updated time of the worker in ms

getToRemoveBlocks

public List<Long> getToRemoveBlocks()

You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.BLOCKS specified. A shared lock is required. This returns a copy so the lock can be released after this method returns.

Returns:

ids of blocks the worker should remove

getUsedBytes

public long getUsedBytes()

Returns:

used space of the worker in bytes

getStorageTierAssoc

public StorageTierAssoc getStorageTierAssoc()

You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.USAGE specified. A shared lock is required.

Returns:

the storage tier mapping for the worker

getTotalBytesOnTiers

public Map<String,Long> getTotalBytesOnTiers()

You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.USAGE specified. A shared lock is required.

Returns:

the total bytes on each storage tier

getUsedBytesOnTiers

public Map<String,Long> getUsedBytesOnTiers()

You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.USAGE specified. A shared lock is required.

Returns:

the used bytes on each storage tier

getStartTime

public long getStartTime()

No locking required.

Returns:

the start time in milliseconds

isRegistered

public boolean isRegistered()

You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.STATUS specified. A shared lock is required.

Returns:

whether the worker has been registered yet

getFreeBytesOnTiers

public Map<String,Long> getFreeBytesOnTiers()

You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.USAGE specified. A shared lock is required.

Returns:

the free bytes on each storage tier

getLostStorage

public Map<String,List<String>> getLostStorage()

You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.USAGE specified. A shared lock is required. This returns a copy so the lock can be released after the map is returned.

Returns:

the map from tier alias to lost storage paths in this worker

hasLostStorage

public boolean hasLostStorage()

You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.USAGE specified. A shared lock is required.

Returns:

true if this worker has lost storage, false otherwise

toString

public String toString()

Overrides:

toString in class Object

updateLastUpdatedTimeMs

public void updateLastUpdatedTimeMs()

Updates the last updated time of the worker in ms. No locking is required.

updateToRemovedBlock

public void updateToRemovedBlock(boolean add,
                                 long blockId)

Adds or removes a block from the to-be-removed blocks set of the worker. You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.BLOCKS specified. An exclusive lock is required.

Parameters:

add - true if to add, to remove otherwise

blockId - the id of the block to be added or removed

updateCapacityBytes

public void updateCapacityBytes(Map<String,Long> capacityBytesOnTiers)

Sets the capacity of the worker in bytes. You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.USAGE specified. An exclusive lock is required.

Parameters:

capacityBytesOnTiers - used bytes on each storage tier

updateUsedBytes

public void updateUsedBytes(Map<String,Long> usedBytesOnTiers)

Sets the used space of the worker in bytes. You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.USAGE specified. An exclusive lock is required.

Parameters:

usedBytesOnTiers - used bytes on each storage tier

updateUsedBytes

public void updateUsedBytes(String tierAlias,
                            long usedBytesOnTier)

Sets the used space of the worker in bytes. You should lock externally with lockWorkerMeta(EnumSet, boolean) with WorkerMetaLockSection.USAGE specified. An exclusive lock is required.

Parameters:

tierAlias - alias of storage tier

usedBytesOnTier - used bytes on certain storage tier

lockWorkerMeta

public LockResource lockWorkerMeta(EnumSet<WorkerMetaLockSection> lockTypes,
                                   boolean isShared)

Locks the corresponding locks on the metadata groups. See the javadoc for this class for example usage. The locks will be acquired in order and later released in the opposite order. The locks can either be shared or exclusive. The isShared flag will apply to all the locks acquired here. This returns a LockResource which can be managed by a try-finally block. See javadoc for WorkerMetaLock for more details about the internals.

Parameters:

lockTypes - the locks

isShared - if false, the locking is exclusive

Returns:

a LockResource of the WorkerMetaLock