alluxio.client.file

Class FileSystemContext

java.lang.Object

alluxio.client.file.FileSystemContext

All Implemented Interfaces:

Closeable, AutoCloseable

@ThreadSafe
public class FileSystemContext
extends Object
implements Closeable

An object which houses resources and information for performing FileSystem operations. Typically, a single JVM should only need one instance of a FileSystem to connect to Alluxio. The reference to that client object should be shared among threads. A second FileSystemContext object should only be created when a user needs to connect to Alluxio with a different Subject and/or AlluxioConfiguration. FileSystemContext instances should be created sparingly because each instance creates its own thread pools of FileSystemMasterClient and BlockMasterClient which can lead to inefficient use of client machine resources. A FileSystemContext should be closed once the user is done performing operations with Alluxio and no more connections need to be made. Once a FileSystemContext is closed it is preferred that the user of the class create a new instance with create(alluxio.conf.AlluxioConfiguration, java.util.List<java.net.InetSocketAddress>) to create a new context, rather than reinitializing using the init(alluxio.ClientContext, alluxio.master.MasterInquireClient) method. NOTE: Each context maintains a pool of file system master clients that is already thread-safe. Synchronizing FileSystemContext methods could lead to deadlock: thread A attempts to acquire a client when there are no clients left in the pool and blocks holding a lock on the FileSystemContext, when thread B attempts to release a client it owns it is unable to do so, because thread A holds the lock on FileSystemContext.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	FileSystemContext.ClientPoolKey Key for block worker client pools.
static class	FileSystemContext.FileSystemContextFactory FileSystemContextFactory, it can be extended.
static class	FileSystemContext.GetWorkerListType This enum class is used for specifying the type of worker list to get.

Field Summary

Fields

Modifier and Type	Field and Description
static FileSystemContext.FileSystemContextFactory	sFileSystemContextFactory

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	FileSystemContext(AlluxioConfiguration conf, BlockWorker blockWorker, List<InetSocketAddress> masterAddresses) Initializes FileSystemContext ID.

Method Summary

Modifier and Type	Method and Description
CloseableResource<BlockMasterClient>	acquireBlockMasterClientResource() Acquires a block master client resource from the block master client pool.
CloseableResource<BlockWorkerClient>	acquireBlockWorkerClient(WorkerNetAddress workerNetAddress) Acquires a block worker client from the client pools.
CloseableResource<FileSystemMasterClient>	acquireMasterClientResource() Acquires a file system master client from the file system master client pool.
io.netty.channel.Channel	acquireNettyChannel(WorkerNetAddress workerNetAddress) Acquires a netty channel from the channel pools.
FileSystemContextReinitializer.ReinitBlockerResource	blockReinit() Acquires the resource to block reinitialization.
void	close() Closes all the resources associated with the context.
static FileSystemContext	create() Creates a FileSystemContext with an empty subject, default config and a null local block worker.
static FileSystemContext	create(AlluxioConfiguration conf) Creates a FileSystemContext with an empty subject and a null local block worker.
static FileSystemContext	create(AlluxioConfiguration conf, List<InetSocketAddress> masterAddresses) Creates a FileSystemContext with an empty subject , a null local block worker, and the given master addresses.
static FileSystemContext	create(ClientContext clientContext)
static FileSystemContext	create(ClientContext ctx, BlockWorker blockWorker)
static FileSystemContext	create(ClientContext ctx, BlockWorker blockWorker, List<InetSocketAddress> masterAddresses)
static FileSystemContext	create(Subject subject, AlluxioConfiguration conf)
static FileSystemContext	create(Subject subject, MasterInquireClient masterInquireClient, AlluxioConfiguration alluxioConf) This method is provided for testing, use the create(alluxio.conf.AlluxioConfiguration, java.util.List<java.net.InetSocketAddress>) methods.
List<BlockWorkerInfo>	getAllWorkers() Gets the worker information list.
protected ConcurrentHashMap<FileSystemContext.ClientPoolKey,BlockWorkerClientPool>	getBlockWorkerClientPoolMap()
List<BlockWorkerInfo>	getCachedWorkers() Gets the cached live worker information list.
List<BlockWorkerInfo>	getCachedWorkers(FileSystemContext.GetWorkerListType type) Gets the cached worker information list.
ClientContext	getClientContext()
AlluxioConfiguration	getClusterConf()
String	getId()
List<BlockWorkerInfo>	getLiveWorkers() Gets the live worker information list.
List<BlockWorkerInfo>	getLostWorkers() Gets the lost worker information list.
InetSocketAddress	getMasterAddress()
MasterClientContext	getMasterClientContext()
protected ConcurrentHashMap<SocketAddress,NettyChannelPool>	getNettyChannelPools()
WorkerNetAddress	getNodeLocalWorker()
Optional<BlockWorker>	getProcessLocalWorker() Acquires the internal block worker as a gateway for worker internal clients to communicate with the local worker directly without going through external RPC frameworks.
boolean	getUriValidationEnabled()
boolean	hasNodeLocalWorker()
boolean	hasProcessLocalWorker()
protected void	init(ClientContext clientContext, MasterInquireClient masterInquireClient) Initializes the context.
protected void	initContext(ClientContext ctx, MasterInquireClient masterInquireClient)
protected void	reCreateReinitialize(FileSystemContextReinitializer fileSystemContextReinitializer)
void	reinit(boolean updateClusterConf) Closes the context, updates configuration from meta master, then re-initializes the context.
void	releaseNettyChannel(WorkerNetAddress workerNetAddress, io.netty.channel.Channel channel) Releases a netty channel to the channel pools.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

sFileSystemContextFactory

public static FileSystemContext.FileSystemContextFactory sFileSystemContextFactory

Constructor Detail

FileSystemContext

protected FileSystemContext(AlluxioConfiguration conf,
                            @Nullable
                            BlockWorker blockWorker,
                            @Nullable
                            List<InetSocketAddress> masterAddresses)

Initializes FileSystemContext ID.

Parameters:

conf - Alluxio configuration

blockWorker - block worker

Method Detail

create

public static FileSystemContext create(AlluxioConfiguration conf,
                                       List<InetSocketAddress> masterAddresses)

Creates a FileSystemContext with an empty subject , a null local block worker, and the given master addresses.

Parameters:

conf - Alluxio configuration

masterAddresses - the master addresses to use, this addresses will be used across reinitialization

Returns:

an instance of file system context with no subject associated

create

public static FileSystemContext create()

Creates a FileSystemContext with an empty subject, default config and a null local block worker.

Returns:

an instance of file system context with no subject associated

create

public static FileSystemContext create(AlluxioConfiguration conf)

Creates a FileSystemContext with an empty subject and a null local block worker.

Parameters:

conf - Alluxio configuration

Returns:

an instance of file system context with no subject associated

create

public static FileSystemContext create(Subject subject,
                                       AlluxioConfiguration conf)

Parameters:

subject - the parent subject

conf - Alluxio configuration

Returns:

a context

create

public static FileSystemContext create(ClientContext clientContext)

Parameters:

clientContext - the ClientContext containing the subject and configuration

Returns:

the FileSystemContext

create

public static FileSystemContext create(ClientContext ctx,
                                       @Nullable
                                       BlockWorker blockWorker)

Parameters:

ctx - client context

blockWorker - block worker

Returns:

a context

create

public static FileSystemContext create(ClientContext ctx,
                                       @Nullable
                                       BlockWorker blockWorker,
                                       @Nullable
                                       List<InetSocketAddress> masterAddresses)

Parameters:

ctx - client context

blockWorker - block worker

masterAddresses - is non-null then the addresses used to connect to the master

Returns:

a context

create

public static FileSystemContext create(Subject subject,
                                       MasterInquireClient masterInquireClient,
                                       AlluxioConfiguration alluxioConf)

This method is provided for testing, use the create(alluxio.conf.AlluxioConfiguration, java.util.List<java.net.InetSocketAddress>) methods. The returned context object will not be cached automatically.

Parameters:

subject - the parent subject

masterInquireClient - the client to use for determining the master; note that if the context is reset, this client will be replaced with a new masterInquireClient based on the original configuration.

alluxioConf - Alluxio configuration

Returns:

the context

init

protected void init(ClientContext clientContext,
                    MasterInquireClient masterInquireClient)

Initializes the context. Only called in the factory methods.

Parameters:

masterInquireClient - the client to use for determining the master

reCreateReinitialize

protected void reCreateReinitialize(@Nullable
                                    FileSystemContextReinitializer fileSystemContextReinitializer)

initContext

protected void initContext(ClientContext ctx,
                           MasterInquireClient masterInquireClient)

close

public void close()
           throws IOException

Closes all the resources associated with the context. Make sure all the resources are released back to this context before calling this close. After closing the context, all the resources that acquired from this context might fail. Only call this when you are done with using the FileSystem associated with this FileSystemContext.

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException

blockReinit

public FileSystemContextReinitializer.ReinitBlockerResource blockReinit()

Acquires the resource to block reinitialization. If reinitialization is happening, this method will block until reinitialization succeeds or fails, if it fails, a RuntimeException will be thrown explaining the reinitialization's failure and automatically closes the resource. RuntimeException is thrown because this method is called before requiring resources from the context, if reinitialization fails, the resources might be half closed, to prevent resource leaks, we thrown RuntimeException here to force callers to fail since there is no way to recover.

Returns:

the resource

reinit

public void reinit(boolean updateClusterConf)
            throws UnavailableException,
                   IOException

Closes the context, updates configuration from meta master, then re-initializes the context. The reinitializer is not closed, which means the heartbeat thread inside it is not stopped. The reinitializer will be reset with the updated context if reinitialization succeeds, otherwise, the reinitializer is not reset. Blocks until there is no active RPCs.

Parameters:

updateClusterConf - whether cluster level configuration should be updated

Throws:

UnavailableException - when failed to load configuration from master

IOException - when failed to close the context

getId

public String getId()

Returns:

the unique ID of this context

getMasterClientContext

public MasterClientContext getMasterClientContext()

Returns:

the MasterClientContext backing this context

getClientContext

public ClientContext getClientContext()

Returns:

the ClientContext backing this FileSystemContext

getClusterConf

public AlluxioConfiguration getClusterConf()

Returns:

the cluster level configuration backing this FileSystemContext

getMasterAddress

public InetSocketAddress getMasterAddress()
                                   throws UnavailableException

Returns:

the master address

Throws:

UnavailableException - if the master address cannot be determined

getUriValidationEnabled

public boolean getUriValidationEnabled()

Returns:

true if URI validation is enabled

acquireMasterClientResource

public CloseableResource<FileSystemMasterClient> acquireMasterClientResource()

Acquires a file system master client from the file system master client pool. The resource is Closeable.

Returns:

the acquired file system master client resource

acquireBlockMasterClientResource

public CloseableResource<BlockMasterClient> acquireBlockMasterClientResource()

Acquires a block master client resource from the block master client pool. The resource is Closeable.

Returns:

the acquired block master client resource

acquireNettyChannel

public io.netty.channel.Channel acquireNettyChannel(WorkerNetAddress workerNetAddress)
                                             throws IOException

Acquires a netty channel from the channel pools. If there is no available client instance available in the pool, it tries to create a new one. And an exception is thrown if it fails to create a new one.

Parameters:

workerNetAddress - the network address of the channel

Returns:

the acquired netty channel

Throws:

IOException

getNettyChannelPools

protected ConcurrentHashMap<SocketAddress,NettyChannelPool> getNettyChannelPools()

releaseNettyChannel

public void releaseNettyChannel(WorkerNetAddress workerNetAddress,
                                io.netty.channel.Channel channel)

Releases a netty channel to the channel pools.

Parameters:

workerNetAddress - the address of the channel

channel - the channel to release

acquireBlockWorkerClient

public CloseableResource<BlockWorkerClient> acquireBlockWorkerClient(WorkerNetAddress workerNetAddress)
                                                              throws IOException

Acquires a block worker client from the client pools. If there is no available client instance available in the pool, it tries to create a new one. And an exception is thrown if it fails to create a new one.

Parameters:

workerNetAddress - the network address of the channel

Returns:

the acquired block worker resource

Throws:

IOException

hasProcessLocalWorker

public boolean hasProcessLocalWorker()

Returns:

if the current client is embedded in a worker

getProcessLocalWorker

public Optional<BlockWorker> getProcessLocalWorker()

Acquires the internal block worker as a gateway for worker internal clients to communicate with the local worker directly without going through external RPC frameworks.

Returns:

the acquired block worker or null if this client is not interal to a block worker

hasNodeLocalWorker

public boolean hasNodeLocalWorker()
                           throws IOException

Returns:

if there is a local worker running the same machine

Throws:

IOException

getNodeLocalWorker

public WorkerNetAddress getNodeLocalWorker()
                                    throws IOException

Returns:

a local worker running the same machine, or null if none is found

Throws:

IOException

getCachedWorkers

public List<BlockWorkerInfo> getCachedWorkers()
                                       throws IOException

Gets the cached live worker information list. This method is relatively cheap as the result is cached, but may not be up-to-date. If up-to-date worker info list is required, use getLiveWorkers() ()} instead.

Returns:

the info of all block workers eligible for reads and writes

Throws:

IOException

getCachedWorkers

public List<BlockWorkerInfo> getCachedWorkers(FileSystemContext.GetWorkerListType type)
                                       throws IOException

Gets the cached worker information list. This method is relatively cheap as the result is cached, but may not be up-to-date. If up-to-date worker info list is required, use getAllWorkers(), getLiveWorkers() ()} or getLostWorkers() ()} instead.

Parameters:

type - get worker list type

Returns:

the info of all block workers eligible for reads and writes

Throws:

IOException

getLiveWorkers

public List<BlockWorkerInfo> getLiveWorkers()
                                     throws IOException

Gets the live worker information list. This method is more expensive than getCachedWorkers(GetWorkerListType). Used when more up-to-date data is needed.

Returns:

the info of live workers

Throws:

IOException

getLostWorkers

public List<BlockWorkerInfo> getLostWorkers()
                                     throws IOException

Gets the lost worker information list. This method is more expensive than getCachedWorkers(GetWorkerListType). Used when more up-to-date data is needed.

Returns:

the info of lost workers

Throws:

IOException

getAllWorkers

public List<BlockWorkerInfo> getAllWorkers()
                                    throws IOException

Gets the worker information list. This method is more expensive than getCachedWorkers(GetWorkerListType). Used when more up-to-date data is needed.

Returns:

the info of all workers

Throws:

IOException

getBlockWorkerClientPoolMap

protected ConcurrentHashMap<FileSystemContext.ClientPoolKey,BlockWorkerClientPool> getBlockWorkerClientPoolMap()