alluxio.client.file

Class FileSystemContext

java.lang.Object

alluxio.client.file.FileSystemContext

All Implemented Interfaces:

Closeable, AutoCloseable

@ThreadSafe
public class FileSystemContext
extends Object
implements Closeable

An object which houses resources and information for performing FileSystem operations. Typically, a single JVM should only need one instance of a FileSystem to connect to Alluxio. The reference to that client object should be shared among threads. A second FileSystemContext object should only be created when a user needs to connect to Alluxio with a different Subject and/or AlluxioConfiguration. FileSystemContext instances should be created sparingly because each instance creates its own thread pools of FileSystemMasterClient and BlockMasterClient which can lead to inefficient use of client machine resources. A FileSystemContext should be closed once the user is done performing operations with Alluxio and no more connections need to be made. Once a FileSystemContext is closed it is preferred that the user of the class create a new instance with create(alluxio.conf.AlluxioConfiguration, java.util.List<java.net.InetSocketAddress>) to create a new context, rather than reinitializing using the init(alluxio.ClientContext, alluxio.master.MasterInquireClient) method. NOTE: Each context maintains a pool of file system master clients that is already thread-safe. Synchronizing FileSystemContext methods could lead to deadlock: thread A attempts to acquire a client when there are no clients left in the pool and blocks holding a lock on the FileSystemContext, when thread B attempts to release a client it owns it is unable to do so, because thread A holds the lock on FileSystemContext.

Method Summary

Modifier and Type	Method and Description
CloseableResource<BlockMasterClient>	acquireBlockMasterClientResource() Acquires a block master client resource from the block master client pool.
CloseableResource<BlockWorkerClient>	acquireBlockWorkerClient(WorkerNetAddress workerNetAddress) Acquires a block worker client from the client pools.
CloseableResource<FileSystemMasterClient>	acquireMasterClientResource() Acquires a file system master client from the file system master client pool.
FileSystemContextReinitializer.ReinitBlockerResource	blockReinit() Acquires the resource to block reinitialization.
void	close() Closes all the resources associated with the context.
static FileSystemContext	create() Creates a FileSystemContext with an empty subject, default config and a null local block worker.
static FileSystemContext	create(AlluxioConfiguration conf) Creates a FileSystemContext with an empty subject and a null local block worker.
static FileSystemContext	create(AlluxioConfiguration conf, List<InetSocketAddress> masterAddresses) Creates a FileSystemContext with an empty subject , a null local block worker, and the given master addresses.
static FileSystemContext	create(ClientContext clientContext)
static FileSystemContext	create(ClientContext ctx, BlockWorker blockWorker)
static FileSystemContext	create(ClientContext ctx, BlockWorker blockWorker, List<InetSocketAddress> masterAddresses)
static FileSystemContext	create(Subject subject, AlluxioConfiguration conf)
static FileSystemContext	create(Subject subject, MasterInquireClient masterInquireClient, AlluxioConfiguration alluxioConf) This method is provided for testing, use the create(alluxio.conf.AlluxioConfiguration, java.util.List<java.net.InetSocketAddress>) methods.
List<BlockWorkerInfo>	getCachedWorkers() Gets the cached worker information list.
ClientContext	getClientContext()
AlluxioConfiguration	getClusterConf()
String	getId()
InetSocketAddress	getMasterAddress()
MasterClientContext	getMasterClientContext()
WorkerNetAddress	getNodeLocalWorker()
AlluxioConfiguration	getPathConf(AlluxioURI path) The path level configuration is a SpecificPathConfiguration.
Optional<BlockWorker>	getProcessLocalWorker() Acquires the internal block worker as a gateway for worker internal clients to communicate with the local worker directly without going through external RPC frameworks.
BlockLocationPolicy	getReadBlockLocationPolicy(AlluxioConfiguration alluxioConf) Gets the readBlockLocationPolicy.
boolean	getUriValidationEnabled()
BlockLocationPolicy	getWriteBlockLocationPolicy(AlluxioConfiguration alluxioConf) Gets the writeBlockLocationPolicy.
boolean	hasNodeLocalWorker()
boolean	hasProcessLocalWorker()
void	reinit(boolean updateClusterConf, boolean updatePathConf) Closes the context, updates configuration from meta master, then re-initializes the context.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

create

public static FileSystemContext create(AlluxioConfiguration conf,
                                       List<InetSocketAddress> masterAddresses)

Creates a FileSystemContext with an empty subject , a null local block worker, and the given master addresses.

Parameters:

conf - Alluxio configuration

masterAddresses - the master addresses to use, this addresses will be used across reinitialization

Returns:

an instance of file system context with no subject associated

create

public static FileSystemContext create()

Creates a FileSystemContext with an empty subject, default config and a null local block worker.

Returns:

an instance of file system context with no subject associated

create

public static FileSystemContext create(AlluxioConfiguration conf)

Creates a FileSystemContext with an empty subject and a null local block worker.

Parameters:

conf - Alluxio configuration

Returns:

an instance of file system context with no subject associated

create

public static FileSystemContext create(Subject subject,
                                       AlluxioConfiguration conf)

Parameters:

subject - the parent subject

conf - Alluxio configuration

Returns:

a context

create

public static FileSystemContext create(ClientContext clientContext)

Parameters:

clientContext - the ClientContext containing the subject and configuration

Returns:

the FileSystemContext

create

public static FileSystemContext create(ClientContext ctx,
                                       @Nullable
                                       BlockWorker blockWorker)

Parameters:

ctx - client context

blockWorker - block worker

Returns:

a context

create

public static FileSystemContext create(ClientContext ctx,
                                       @Nullable
                                       BlockWorker blockWorker,
                                       @Nullable
                                       List<InetSocketAddress> masterAddresses)

Parameters:

ctx - client context

blockWorker - block worker

masterAddresses - is non-null then the addresses used to connect to the master

Returns:

a context

create

public static FileSystemContext create(Subject subject,
                                       MasterInquireClient masterInquireClient,
                                       AlluxioConfiguration alluxioConf)

This method is provided for testing, use the create(alluxio.conf.AlluxioConfiguration, java.util.List<java.net.InetSocketAddress>) methods. The returned context object will not be cached automatically.

Parameters:

subject - the parent subject

masterInquireClient - the client to use for determining the master; note that if the context is reset, this client will be replaced with a new masterInquireClient based on the original configuration.

alluxioConf - Alluxio configuration

Returns:

the context

close

public void close()
           throws IOException

Closes all the resources associated with the context. Make sure all the resources are released back to this context before calling this close. After closing the context, all the resources that acquired from this context might fail. Only call this when you are done with using the FileSystem associated with this FileSystemContext.

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException

blockReinit

public FileSystemContextReinitializer.ReinitBlockerResource blockReinit()

Acquires the resource to block reinitialization. If reinitialization is happening, this method will block until reinitialization succeeds or fails, if it fails, a RuntimeException will be thrown explaining the reinitialization's failure and automatically closes the resource. RuntimeException is thrown because this method is called before requiring resources from the context, if reinitialization fails, the resources might be half closed, to prevent resource leaks, we thrown RuntimeException here to force callers to fail since there is no way to recover.

Returns:

the resource

reinit

public void reinit(boolean updateClusterConf,
                   boolean updatePathConf)
            throws UnavailableException,
                   IOException

Closes the context, updates configuration from meta master, then re-initializes the context. The reinitializer is not closed, which means the heartbeat thread inside it is not stopped. The reinitializer will be reset with the updated context if reinitialization succeeds, otherwise, the reinitializer is not reset. Blocks until there is no active RPCs.

Parameters:

updateClusterConf - whether cluster level configuration should be updated

updatePathConf - whether path level configuration should be updated

Throws:

UnavailableException - when failed to load configuration from master

IOException - when failed to close the context

getId

public String getId()

Returns:

the unique ID of this context

getMasterClientContext

public MasterClientContext getMasterClientContext()

Returns:

the MasterClientContext backing this context

getClientContext

public ClientContext getClientContext()

Returns:

the ClientContext backing this FileSystemContext

getClusterConf

public AlluxioConfiguration getClusterConf()

Returns:

the cluster level configuration backing this FileSystemContext

getPathConf

public AlluxioConfiguration getPathConf(AlluxioURI path)

The path level configuration is a SpecificPathConfiguration. If path level configuration has never been loaded from meta master yet, it will be loaded.

Parameters:

path - the path to get the configuration for

Returns:

the path level configuration for the specific path

getMasterAddress

public InetSocketAddress getMasterAddress()
                                   throws UnavailableException

Returns:

the master address

Throws:

UnavailableException - if the master address cannot be determined

getUriValidationEnabled

public boolean getUriValidationEnabled()

Returns:

true if URI validation is enabled

acquireMasterClientResource

public CloseableResource<FileSystemMasterClient> acquireMasterClientResource()

Acquires a file system master client from the file system master client pool. The resource is Closeable.

Returns:

the acquired file system master client resource

acquireBlockMasterClientResource

public CloseableResource<BlockMasterClient> acquireBlockMasterClientResource()

Acquires a block master client resource from the block master client pool. The resource is Closeable.

Returns:

the acquired block master client resource

acquireBlockWorkerClient

public CloseableResource<BlockWorkerClient> acquireBlockWorkerClient(WorkerNetAddress workerNetAddress)
                                                              throws IOException

Acquires a block worker client from the client pools. If there is no available client instance available in the pool, it tries to create a new one. And an exception is thrown if it fails to create a new one.

Parameters:

workerNetAddress - the network address of the channel

Returns:

the acquired block worker resource

Throws:

IOException

hasProcessLocalWorker

public boolean hasProcessLocalWorker()

Returns:

if the current client is embedded in a worker

getProcessLocalWorker

public Optional<BlockWorker> getProcessLocalWorker()

Acquires the internal block worker as a gateway for worker internal clients to communicate with the local worker directly without going through external RPC frameworks.

Returns:

the acquired block worker or null if this client is not interal to a block worker

hasNodeLocalWorker

public boolean hasNodeLocalWorker()
                           throws IOException

Returns:

if there is a local worker running the same machine

Throws:

IOException

getNodeLocalWorker

public WorkerNetAddress getNodeLocalWorker()
                                    throws IOException

Returns:

a local worker running the same machine, or null if none is found

Throws:

IOException

getCachedWorkers

public List<BlockWorkerInfo> getCachedWorkers()
                                       throws IOException

Gets the cached worker information list. This method is relatively cheap as the result is cached, but may not be up-to-date. If up-to-date worker info list is required, use getAllWorkers() instead.

Returns:

the info of all block workers eligible for reads and writes

Throws:

IOException

getReadBlockLocationPolicy

public BlockLocationPolicy getReadBlockLocationPolicy(AlluxioConfiguration alluxioConf)

Gets the readBlockLocationPolicy.

Parameters:

alluxioConf - Alluxio configuration

Returns:

the readBlockLocationPolicy

getWriteBlockLocationPolicy

public BlockLocationPolicy getWriteBlockLocationPolicy(AlluxioConfiguration alluxioConf)

Gets the writeBlockLocationPolicy.

Parameters:

alluxioConf - Alluxio configuration

Returns:

the writeBlockLocationPolicy