alluxio.worker.block

Class DefaultBlockWorker

java.lang.Object

alluxio.worker.AbstractWorker

alluxio.worker.block.DefaultBlockWorker

All Implemented Interfaces:

Server<WorkerNetAddress>, BlockWorker, SessionCleanable, Worker

Direct Known Subclasses:

AllMasterRegistrationBlockWorker

@NotThreadSafe
public class DefaultBlockWorker
extends AbstractWorker
implements BlockWorker

The class is responsible for managing all top level components of the Block Worker. This includes: Periodic Threads: BlockMasterSync (Worker to Master continuous communication) Logic: DefaultBlockWorker (Logic for all block related storage operations)

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	DefaultBlockWorker.Metrics This class contains some metrics related to the block worker.
class	DefaultBlockWorker.StorageChecker StorageChecker periodically checks the health of each storage path and report missing blocks to BlockWorker.

Field Summary

Fields

Modifier and Type	Field and Description
protected WorkerNetAddress	mAddress
protected BlockStore	mBlockStore Block Store manager.
protected FileSystemMasterClient	mFileSystemMasterClient Client for all file system master communication.
protected com.google.common.io.Closer	mResourceCloser Used to close resources during stop.
protected AtomicReference<Long>	mWorkerId The worker ID for this worker.

Constructor Summary

Constructors

Constructor and Description
DefaultBlockWorker(BlockMasterClientPool blockMasterClientPool, FileSystemMasterClient fileSystemMasterClient, Sessions sessions, BlockStore blockStore, AtomicReference<Long> workerId) Constructs a default block worker.

Method Summary

Modifier and Type	Method and Description
void	abortBlock(long sessionId, long blockId) Aborts the temporary block created by the session.
void	askForWorkerId(WorkerNetAddress address) Ask the master for a workerId.
void	asyncCache(AsyncCacheRequest request) Deprecated.
void	cache(CacheRequest request) Submits the cache request to cache manager to execute.
void	cleanupSession(long sessionId) Cleans up the session.
void	clearMetrics() Clears the worker metrics.
void	commitBlock(long sessionId, long blockId, boolean pinOnCreate) Commits a block to Alluxio managed space.
void	commitBlockInUfs(long blockId, long length) Commits a block in UFS.
String	createBlock(long sessionId, long blockId, int tier, CreateBlockOptions createBlockOptions) Creates a block in Alluxio managed space.
BlockReader	createBlockReader(long sessionId, long blockId, long offset, boolean positionShort, Protocol.OpenUfsBlockOptions options) Closes a UFS block for a client session.
BlockWriter	createBlockWriter(long sessionId, long blockId) Creates a BlockWriter for an existing temporary block which is already created by BlockWorker.createBlock(long, long, int, alluxio.worker.block.CreateBlockOptions).
BlockReader	createUfsBlockReader(long sessionId, long blockId, long offset, boolean positionShort, Protocol.OpenUfsBlockOptions options) Creates a block reader to read a UFS block starting from given block offset.
void	freeWorker() Frees all blocks in the current worker by deleting all block store directories.
BlockStore	getBlockStore() get the LocalBlockStore that manages local blocks.
Configuration	getConfiguration(GetConfigurationPOptions options)
Set<Class<? extends Server>>	getDependencies()
FileInfo	getFileInfo(long fileId) Gets the file information.
String	getName()
BlockHeartbeatReport	getReport() Gets a report for the periodic heartbeat to master.
Map<ServiceType,GrpcService>	getServices()
BlockStoreMeta	getStoreMeta() Gets the metadata for the entire block store.
BlockStoreMeta	getStoreMetaFull() Similar as BlockWorker.getStoreMeta() except that this also contains full blockId list.
List<String>	getWhiteList()
WorkerNetAddress	getWorkerAddress()
AtomicReference<Long>	getWorkerId()
CompletableFuture<List<BlockStatus>>	load(List<Block> blocks, UfsReadOptions options) Load blocks into alluxio.
void	removeBlock(long sessionId, long blockId) Frees a block from Alluxio managed space.
void	requestSpace(long sessionId, long blockId, long additionalBytes) Request an amount of space for a block in its storage directory.
protected void	setupBlockMasterSync()
void	start(WorkerNetAddress address) Runs the block worker.
void	stop() Stops the block worker.
void	updatePinList(Set<Long> pinnedInodes) Sets the pinlist for the underlying block store.

Methods inherited from class alluxio.worker.AbstractWorker

close, getExecutorService

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface alluxio.Server

close

Field Detail

mResourceCloser

protected final com.google.common.io.Closer mResourceCloser

Used to close resources during stop.

mFileSystemMasterClient

protected final FileSystemMasterClient mFileSystemMasterClient

Client for all file system master communication.

mBlockStore

protected final BlockStore mBlockStore

Block Store manager.

mWorkerId

protected final AtomicReference<Long> mWorkerId

The worker ID for this worker. This is initialized in start(WorkerNetAddress) and may be updated by the block sync thread if the master requests re-registration.

mAddress

protected WorkerNetAddress mAddress

Constructor Detail

DefaultBlockWorker

public DefaultBlockWorker(BlockMasterClientPool blockMasterClientPool,
                          FileSystemMasterClient fileSystemMasterClient,
                          Sessions sessions,
                          BlockStore blockStore,
                          AtomicReference<Long> workerId)

Constructs a default block worker.

Parameters:

blockMasterClientPool - a client pool for talking to the block master

fileSystemMasterClient - a client for talking to the file system master

sessions - an object for tracking and cleaning up client sessions

blockStore - an Alluxio block store

workerId - worker id

Method Detail

getBlockStore

public BlockStore getBlockStore()

get the LocalBlockStore that manages local blocks.

Specified by:

getBlockStore in interface BlockWorker

Returns:

the LocalBlockStore that manages local blocks

getWorkerAddress

public WorkerNetAddress getWorkerAddress()

Specified by:

getWorkerAddress in interface BlockWorker

Returns:

the worker address

getDependencies

public Set<Class<? extends Server>> getDependencies()

Specified by:

getDependencies in interface Server<WorkerNetAddress>

Returns:

the server dependencies

getName

public String getName()

Specified by:

getName in interface Server<WorkerNetAddress>

Returns:

the server's name

getServices

public Map<ServiceType,GrpcService> getServices()

Specified by:

getServices in interface Server<WorkerNetAddress>

Returns:

a map from service names to gRPC serviced that serve RPCs for this server

getWorkerId

public AtomicReference<Long> getWorkerId()

Specified by:

getWorkerId in interface BlockWorker

Returns:

the worker id

start

public void start(WorkerNetAddress address)
           throws IOException

Runs the block worker. The thread must be called after all services (e.g., web, dataserver) started. BlockWorker doesn't support being restarted!

Specified by:

start in interface Server<WorkerNetAddress>

Overrides:

start in class AbstractWorker

Parameters:

address - the start options

Throws:

IOException

setupBlockMasterSync

protected void setupBlockMasterSync()
                             throws IOException

Throws:

IOException

askForWorkerId

public void askForWorkerId(WorkerNetAddress address)

Ask the master for a workerId. Should not be called outside of testing

Parameters:

address - the address this worker operates on

stop

public void stop()
          throws IOException

Stops the block worker. This method should only be called to terminate the worker. BlockWorker doesn't support being restarted!

Specified by:

stop in interface Server<WorkerNetAddress>

Overrides:

stop in class AbstractWorker

Throws:

IOException

abortBlock

public void abortBlock(long sessionId,
                       long blockId)
                throws IOException

Description copied from interface: BlockWorker

Aborts the temporary block created by the session.

Specified by:

abortBlock in interface BlockWorker

Parameters:

sessionId - the id of the client

blockId - the id of the block to be aborted

Throws:

IOException

commitBlock

public void commitBlock(long sessionId,
                        long blockId,
                        boolean pinOnCreate)

Description copied from interface: BlockWorker

Commits a block to Alluxio managed space. The block must be temporary. The block will not be persisted or accessible before commitBlock succeeds.

Specified by:

commitBlock in interface BlockWorker

Parameters:

sessionId - the id of the client

blockId - the id of the block to commit

pinOnCreate - whether to pin block on create

commitBlockInUfs

public void commitBlockInUfs(long blockId,
                             long length)

Description copied from interface: BlockWorker

Commits a block in UFS.

Specified by:

commitBlockInUfs in interface BlockWorker

Parameters:

blockId - the id of the block to commit

length - length of the block to commit

createBlock

public String createBlock(long sessionId,
                          long blockId,
                          int tier,
                          CreateBlockOptions createBlockOptions)

Description copied from interface: BlockWorker

Creates a block in Alluxio managed space. Calls BlockWorker.createBlockWriter(long, long) to get a writer for writing to the block. The block will be temporary until it is committed by BlockWorker.commitBlock(long, long, boolean) . Throws an IllegalArgumentException if the location does not belong to tiered storage.

Specified by:

createBlock in interface BlockWorker

Parameters:

sessionId - the id of the client

blockId - the id of the block to create

tier - the tier to place the new block in BlockStoreLocation.ANY_TIER for any tier

createBlockOptions - the createBlockOptions

Returns:

a string representing the path to the local file

createBlockWriter

public BlockWriter createBlockWriter(long sessionId,
                                     long blockId)
                              throws IOException

Description copied from interface: BlockWorker

Creates a BlockWriter for an existing temporary block which is already created by BlockWorker.createBlock(long, long, int, alluxio.worker.block.CreateBlockOptions).

Specified by:

createBlockWriter in interface BlockWorker

Parameters:

sessionId - the id of the client

blockId - the id of the block to be opened for writing

Returns:

the block writer for the local block file

Throws:

IOException

getReport

public BlockHeartbeatReport getReport()

Description copied from interface: BlockWorker

Gets a report for the periodic heartbeat to master. Contains the blocks added since the last heart beat and blocks removed since the last heartbeat.

Specified by:

getReport in interface BlockWorker

Returns:

a block heartbeat report

getStoreMeta

public BlockStoreMeta getStoreMeta()

Description copied from interface: BlockWorker

Gets the metadata for the entire block store. Contains the block mapping per storage dir and the total capacity and used capacity of each tier. This function is cheap.

Specified by:

getStoreMeta in interface BlockWorker

Returns:

the block store metadata

getStoreMetaFull

public BlockStoreMeta getStoreMetaFull()

Description copied from interface: BlockWorker

Similar as BlockWorker.getStoreMeta() except that this also contains full blockId list. This function is expensive.

Specified by:

getStoreMetaFull in interface BlockWorker

Returns:

the full block store metadata

getWhiteList

public List<String> getWhiteList()

Specified by:

getWhiteList in interface BlockWorker

Returns:

the white list

createUfsBlockReader

public BlockReader createUfsBlockReader(long sessionId,
                                        long blockId,
                                        long offset,
                                        boolean positionShort,
                                        Protocol.OpenUfsBlockOptions options)
                                 throws IOException

Description copied from interface: BlockWorker

Creates a block reader to read a UFS block starting from given block offset. Owner of this block reader must close it to cleanup state.

Specified by:

createUfsBlockReader in interface BlockWorker

Parameters:

sessionId - the client session ID

blockId - the ID of the UFS block to read

offset - the offset within the block

positionShort - whether the operation is using positioned read to a small buffer size

options - the options

Returns:

the block reader instance

Throws:

IOException - if it fails to get block reader

removeBlock

public void removeBlock(long sessionId,
                        long blockId)
                 throws IOException

Description copied from interface: BlockWorker

Frees a block from Alluxio managed space.

Specified by:

removeBlock in interface BlockWorker

Parameters:

sessionId - the id of the client

blockId - the id of the block to be freed

Throws:

IOException

freeWorker

public void freeWorker()
                throws IOException

Description copied from interface: BlockWorker

Frees all blocks in the current worker by deleting all block store directories. Whether this method returns successfully or exceptionally, the worker should not be used in any way.

Specified by:

freeWorker in interface BlockWorker

Throws:

IOException - if free fails

requestSpace

public void requestSpace(long sessionId,
                         long blockId,
                         long additionalBytes)

Description copied from interface: BlockWorker

Request an amount of space for a block in its storage directory. The block must be a temporary block.

Specified by:

requestSpace in interface BlockWorker

Parameters:

sessionId - the id of the client

blockId - the id of the block to allocate space to

additionalBytes - the amount of bytes to allocate

asyncCache

@Deprecated
public void asyncCache(AsyncCacheRequest request)

Deprecated.

Description copied from interface: BlockWorker

Submits the async cache request to async cache manager to execute.

Specified by:

asyncCache in interface BlockWorker

Parameters:

request - the async cache request

cache

public void cache(CacheRequest request)
           throws AlluxioException,
                  IOException

Description copied from interface: BlockWorker

Submits the cache request to cache manager to execute.

Specified by:

cache in interface BlockWorker

Parameters:

request - the cache request

Throws:

AlluxioException

IOException

load

public CompletableFuture<List<BlockStatus>> load(List<Block> blocks,
                                                 UfsReadOptions options)

Description copied from interface: BlockWorker

Load blocks into alluxio.

Specified by:

load in interface BlockWorker

Parameters:

blocks - list of fileBlocks, one file blocks contains blocks belong to one file

options - read ufs options

Returns:

future of load status for failed blocks

updatePinList

public void updatePinList(Set<Long> pinnedInodes)

Description copied from interface: BlockWorker

Sets the pinlist for the underlying block store.

Specified by:

updatePinList in interface BlockWorker

Parameters:

pinnedInodes - a set of pinned inodes

getFileInfo

public FileInfo getFileInfo(long fileId)
                     throws IOException

Description copied from interface: BlockWorker

Gets the file information.

Specified by:

getFileInfo in interface BlockWorker

Parameters:

fileId - the file id

Returns:

the file info

Throws:

IOException

createBlockReader

public BlockReader createBlockReader(long sessionId,
                                     long blockId,
                                     long offset,
                                     boolean positionShort,
                                     Protocol.OpenUfsBlockOptions options)
                              throws IOException

Closes a UFS block for a client session. It also commits the block to Alluxio block store if the UFS block has been cached successfully.

Specified by:

createBlockReader in interface BlockWorker

Parameters:

sessionId - the session ID

blockId - the block ID

offset - the offset within the block

positionShort - whether the operation is using positioned read to a small buffer size

options - the options

Returns:

a block reader to read data from

Throws:

IOException - if it fails to get block reader

clearMetrics

public void clearMetrics()

Description copied from interface: BlockWorker

Clears the worker metrics.

Specified by:

clearMetrics in interface BlockWorker

getConfiguration

public Configuration getConfiguration(GetConfigurationPOptions options)

Specified by:

getConfiguration in interface BlockWorker

Parameters:

options - method options

Returns:

configuration information list

cleanupSession

public void cleanupSession(long sessionId)

Description copied from interface: SessionCleanable

Cleans up the session.

Specified by:

cleanupSession in interface SessionCleanable

Parameters:

sessionId - the session id