alluxio.worker.block

Class BlockMasterClient

java.lang.Object

alluxio.AbstractClient

alluxio.AbstractMasterClient

alluxio.worker.block.BlockMasterClient

All Implemented Interfaces:

Client, Closeable, AutoCloseable

Direct Known Subclasses:

CachingBlockMasterClient

@ThreadSafe
public class BlockMasterClient
extends AbstractMasterClient

A wrapper for the gRPC client to interact with the block master, used by alluxio worker.

Nested Class Summary

Nested classes/interfaces inherited from class alluxio.AbstractClient

AbstractClient.RpcCallable<V>

Field Summary

Fields

Modifier and Type	Field and Description
BlockMasterWorkerServiceGrpc.BlockMasterWorkerServiceStub	mAsyncClient
BlockMasterWorkerServiceGrpc.BlockMasterWorkerServiceBlockingStub	mClient

Fields inherited from class alluxio.AbstractClient

mAddress, mChannel, mClosed, mConnected, mContext, mServiceVersion, mVersionService

Constructor Summary

Constructors

Constructor and Description
BlockMasterClient(MasterClientContext conf) Creates a new instance of BlockMasterClient for the worker.

Method Summary

Modifier and Type	Method and Description
void	acquireRegisterLeaseWithBackoff(long workerId, int estimatedBlockCount, RetryPolicy retry) Acquires a RegisterLease from the master with the RetryPolicy specified.
protected void	afterConnect() This method is called after the connection is made to the remote.
void	commitBlock(long workerId, long usedBytesOnTier, String tierAlias, String mediumType, long blockId, long length) Commits a block on a worker.
void	commitBlockInUfs(long blockId, long length) Commits a block in Ufs.
List<LocationBlockIdListEntry>	convertBlockListMapToProto(Map<BlockStoreLocation,List<Long>> blockListOnLocation) Converts the block list map to a proto list.
long	getId(WorkerNetAddress address) Returns a worker id for a workers net address.
protected ServiceType	getRemoteServiceType()
protected String	getServiceName()
protected long	getServiceVersion()
Command	heartbeat(long workerId, Map<String,Long> capacityBytesOnTiers, Map<String,Long> usedBytesOnTiers, List<Long> removedBlocks, Map<BlockStoreLocation,List<Long>> addedBlocks, Map<String,List<String>> lostStorage, List<Metric> metrics) The method the worker should periodically execute to heartbeat back to the master.
void	register(long workerId, List<String> storageTierAliases, Map<String,Long> totalBytesOnTiers, Map<String,Long> usedBytesOnTiers, Map<BlockStoreLocation,List<Long>> currentBlocksOnLocation, Map<String,List<String>> lostStorage, List<ConfigProperty> configList) The method the worker should execute to register with the block master.
void	registerWithStream(long workerId, List<String> storageTierAliases, Map<String,Long> totalBytesOnTiers, Map<String,Long> usedBytesOnTiers, Map<BlockStoreLocation,List<Long>> currentBlocksOnLocation, Map<String,List<String>> lostStorage, List<ConfigProperty> configList) Registers with the master in a stream.

Methods inherited from class alluxio.AbstractMasterClient

getAddress, getConfAddress

Methods inherited from class alluxio.AbstractClient

afterDisconnect, beforeConnect, beforeDisconnect, checkVersion, close, connect, disconnect, getRemoteServiceVersion, isClosed, isConnected, retryRPC, retryRPC

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

mClient

public BlockMasterWorkerServiceGrpc.BlockMasterWorkerServiceBlockingStub mClient

mAsyncClient

public BlockMasterWorkerServiceGrpc.BlockMasterWorkerServiceStub mAsyncClient

Constructor Detail

BlockMasterClient

public BlockMasterClient(MasterClientContext conf)

Creates a new instance of BlockMasterClient for the worker.

Parameters:

conf - master client configuration

Method Detail

getRemoteServiceType

protected ServiceType getRemoteServiceType()

Specified by:

getRemoteServiceType in class AbstractClient

Returns:

the type of remote service

getServiceName

protected String getServiceName()

Specified by:

getServiceName in class AbstractClient

Returns:

a string representing the specific service

getServiceVersion

protected long getServiceVersion()

Specified by:

getServiceVersion in class AbstractClient

Returns:

the client service version

afterConnect

protected void afterConnect()
                     throws IOException

Description copied from class: AbstractClient

This method is called after the connection is made to the remote. Implementations should create internal state to finish the connection process.

Overrides:

afterConnect in class AbstractClient

Throws:

IOException

commitBlock

public void commitBlock(long workerId,
                        long usedBytesOnTier,
                        String tierAlias,
                        String mediumType,
                        long blockId,
                        long length)
                 throws IOException

Commits a block on a worker.

Parameters:

workerId - the worker id committing the block

usedBytesOnTier - the amount of used bytes on the tier the block is committing to

tierAlias - the alias of the tier the block is being committed to

mediumType - the medium type the block is being committed to

blockId - the block id being committed

length - the length of the block being committed

Throws:

IOException

commitBlockInUfs

public void commitBlockInUfs(long blockId,
                             long length)
                      throws IOException

Commits a block in Ufs.

Parameters:

blockId - the block id being committed

length - the length of the block being committed

Throws:

IOException

getId

public long getId(WorkerNetAddress address)
           throws IOException

Returns a worker id for a workers net address.

Parameters:

address - the net address to get a worker id for

Returns:

a worker id

Throws:

IOException

convertBlockListMapToProto

public List<LocationBlockIdListEntry> convertBlockListMapToProto(Map<BlockStoreLocation,List<Long>> blockListOnLocation)

Converts the block list map to a proto list. Because the list is flattened from a map, in the list no two LocationBlockIdListEntry instances shall have the same BlockStoreLocationProto. The uniqueness of BlockStoreLocationProto is determined by tier alias and medium type. That means directories with the same tier alias and medium type will be merged into the same LocationBlockIdListEntry.

Parameters:

blockListOnLocation - a map from block location to the block list

Returns:

a flattened and deduplicated list

heartbeat

public Command heartbeat(long workerId,
                         Map<String,Long> capacityBytesOnTiers,
                         Map<String,Long> usedBytesOnTiers,
                         List<Long> removedBlocks,
                         Map<BlockStoreLocation,List<Long>> addedBlocks,
                         Map<String,List<String>> lostStorage,
                         List<Metric> metrics)
                  throws IOException

The method the worker should periodically execute to heartbeat back to the master.

Parameters:

workerId - the worker id

capacityBytesOnTiers - a mapping from storage tier alias to capacity bytes

usedBytesOnTiers - a mapping from storage tier alias to used bytes

removedBlocks - a list of block removed from this worker

addedBlocks - a mapping from storage tier alias to added blocks

lostStorage - a mapping from storage tier alias to a list of lost storage paths

metrics - a list of worker metrics

Returns:

an optional command for the worker to execute

Throws:

IOException

acquireRegisterLeaseWithBackoff

public void acquireRegisterLeaseWithBackoff(long workerId,
                                            int estimatedBlockCount,
                                            RetryPolicy retry)
                                     throws IOException,
                                            FailedToAcquireRegisterLeaseException

Acquires a RegisterLease from the master with the RetryPolicy specified. If all the retry attempts have been exhaused, a FailedToAcquireRegisterLeaseException will be thrown.

Parameters:

workerId - the worker ID

estimatedBlockCount - the number of blocks this worker currently holds There is a gap between acquiring a lease and generating the RegisterWorkerPRequest so the real block count may be different. But this is a good hint for the master to guess how much resource the registration will take.

retry - a retry policy

Throws:

IOException

FailedToAcquireRegisterLeaseException

register

public void register(long workerId,
                     List<String> storageTierAliases,
                     Map<String,Long> totalBytesOnTiers,
                     Map<String,Long> usedBytesOnTiers,
                     Map<BlockStoreLocation,List<Long>> currentBlocksOnLocation,
                     Map<String,List<String>> lostStorage,
                     List<ConfigProperty> configList)
              throws IOException

The method the worker should execute to register with the block master.

Parameters:

workerId - the worker id of the worker registering

storageTierAliases - a list of storage tier aliases in ordinal order

totalBytesOnTiers - mapping from storage tier alias to total bytes

usedBytesOnTiers - mapping from storage tier alias to used bytes

currentBlocksOnLocation - mapping from storage tier alias to the list of list of blocks

lostStorage - mapping from storage tier alias to the list of lost storage paths

configList - a list of configurations

Throws:

IOException

registerWithStream

public void registerWithStream(long workerId,
                               List<String> storageTierAliases,
                               Map<String,Long> totalBytesOnTiers,
                               Map<String,Long> usedBytesOnTiers,
                               Map<BlockStoreLocation,List<Long>> currentBlocksOnLocation,
                               Map<String,List<String>> lostStorage,
                               List<ConfigProperty> configList)
                        throws IOException

Registers with the master in a stream.

Parameters:

workerId - the worker ID

storageTierAliases - storage/tier setup from the configuration

totalBytesOnTiers - the capacity of each tier

usedBytesOnTiers - the current usage of each tier

currentBlocksOnLocation - the blocks in each tier/dir

lostStorage - the lost storage paths

configList - the configuration properties

Throws:

IOException