alluxio.client.file

Class BaseFileSystem

java.lang.Object

alluxio.client.file.BaseFileSystem

All Implemented Interfaces:

FileSystem, Closeable, AutoCloseable

@ThreadSafe
public class BaseFileSystem
extends Object
implements FileSystem

Default implementation of the FileSystem interface. Developers can extend this class instead of implementing the interface. This implementation reads and writes data through FileInStream and FileOutStream. This class is thread safe.

Nested Class Summary

Nested classes/interfaces inherited from interface alluxio.client.file.FileSystem

FileSystem.Factory

Field Summary

Fields

Modifier and Type	Field and Description
protected BlockStoreClient	mBlockStore
protected boolean	mClosed
protected FileSystemContext	mFsContext
protected static Error	UNREACHABLE_CODE_ERROR

Constructor Summary

Constructors

Constructor and Description
BaseFileSystem(FileSystemContext fsContext) Constructs a new base file system.

Method Summary

Modifier and Type	Method and Description
void	checkAccess(AlluxioURI path, CheckAccessPOptions options) Checks access to a path.
protected void	checkUri(AlluxioURI uri) Checks an AlluxioURI for scheme and authority information.
void	close() Shuts down the FileSystem.
void	createDirectory(AlluxioURI path, CreateDirectoryPOptions options) Creates a directory.
FileOutStream	createFile(AlluxioURI path, CreateFilePOptions options) Creates a file.
void	delete(AlluxioURI path, DeletePOptions options) Deletes a file or a directory.
boolean	exists(AlluxioURI path, ExistsPOptions options) Checks whether a path exists in Alluxio space.
void	free(AlluxioURI path, FreePOptions options) Evicts any data under the given path from Alluxio space, but does not delete the data from the UFS.
List<BlockLocationInfo>	getBlockLocations(URIStatus status) Builds a list of BlockLocationInfo for the given file.
AlluxioConfiguration	getConf()
FileSystemContext	getFileSystemContext()
String	getJobProgress(JobDescription jobDescription, JobProgressReportFormat format, boolean verbose) Get progress of a job.
Map<String,MountPointInfo>	getMountTable(boolean checkUfs) Lists all mount points and their corresponding under storage addresses.
URIStatus	getStatus(AlluxioURI path, GetStatusPOptions options) Gets the URIStatus object that represents the metadata of an Alluxio path.
List<SyncPointInfo>	getSyncPathList() Lists all the actively synced paths.
boolean	isClosed() If there are operations currently running and close is called concurrently the behavior is undefined.
void	iterateStatus(AlluxioURI path, ListStatusPOptions options, java.util.function.Consumer<? super URIStatus> action) Performs a specific action on each URIStatus in the result of FileSystem.listStatus(alluxio.AlluxioURI).
List<URIStatus>	listStatus(AlluxioURI path, ListStatusPOptions options) If the path is a directory, returns the URIStatus of all the direct entries in it.
ListStatusPartialResult	listStatusPartial(AlluxioURI path, ListStatusPartialPOptions options) Same as FileSystem.listStatus(AlluxioURI, ListStatusPOptions) except may only return a subset of the results as determined by the options parameter.
void	loadMetadata(AlluxioURI path, ListStatusPOptions options) Loads metadata about a path in the UFS to Alluxio.
void	mount(AlluxioURI alluxioPath, AlluxioURI ufsPath, MountPOptions options) Mounts a UFS subtree to the given Alluxio path.
void	needsSync(AlluxioURI path) Marks the path in Alluxio as needing sync with the UFS.
FileInStream	openFile(AlluxioURI path, OpenFilePOptions options) Opens a file for reading.
FileInStream	openFile(URIStatus status, OpenFilePOptions options) Opens a file for reading.
void	persist(AlluxioURI path, ScheduleAsyncPersistencePOptions options) Schedules the given path to be asynchronously persisted to the under file system.
void	rename(AlluxioURI src, AlluxioURI dst, RenamePOptions options) Renames an existing Alluxio path to another Alluxio path in Alluxio.
AlluxioURI	reverseResolve(AlluxioURI ufsUri) Reverse resolve a ufs uri.
void	setAcl(AlluxioURI path, SetAclAction action, List<AclEntry> entries, SetAclPOptions options) Sets the ACL for a path.
void	setAttribute(AlluxioURI path, SetAttributePOptions options) Sets any number of a path's attributes, such as TTL and pin status.
void	startSync(AlluxioURI path) Starts the active syncing process on an Alluxio path.
boolean	stopJob(JobDescription jobDescription) Stop a job in scheduler.
void	stopSync(AlluxioURI path) Stops the active syncing process on an Alluxio path.
Optional<String>	submitJob(JobRequest jobRequest) Submit a job to scheduler.
void	unmount(AlluxioURI path, UnmountPOptions options) Unmounts a UFS subtree identified by the given Alluxio path.
void	updateMount(AlluxioURI alluxioPath, MountPOptions options) Updates the options for an existing mount point.
protected void	wrapAndThrowAlluxioStatusException(AlluxioStatusException e)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface alluxio.client.file.FileSystem

createDirectory, createFile, delete, exists, free, getBlockLocations, getMountTable, getStatus, iterateStatus, listStatus, loadMetadata, mount, openFile, persist, rename, setAcl, setAttribute, unmount

Field Detail

mFsContext

protected final FileSystemContext mFsContext

mBlockStore

protected final BlockStoreClient mBlockStore

mClosed

protected volatile boolean mClosed

UNREACHABLE_CODE_ERROR

protected static final Error UNREACHABLE_CODE_ERROR

Constructor Detail

BaseFileSystem

public BaseFileSystem(FileSystemContext fsContext)

Constructs a new base file system.

Parameters:

fsContext - file system context

Method Detail

close

public void close()
           throws IOException

Shuts down the FileSystem. Closes all thread pools and resources used to perform operations. If any operations are called after closing the context the behavior is undefined.

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException

isClosed

public boolean isClosed()

Description copied from interface: FileSystem

If there are operations currently running and close is called concurrently the behavior is undefined. After closing a FileSystem, any operations that are performed result in undefined behavior.

Specified by:

isClosed in interface FileSystem

Returns:

whether or not this FileSystem has been closed

checkAccess

public void checkAccess(AlluxioURI path,
                        CheckAccessPOptions options)
                 throws InvalidPathException,
                        IOException,
                        AlluxioException

Description copied from interface: FileSystem

Checks access to a path.

Specified by:

checkAccess in interface FileSystem

Parameters:

path - the path of the directory to create in Alluxio space

options - options to associate with this operation

Throws:

InvalidPathException - if the path is invalid

AccessControlException - if the access is denied

IOException

AlluxioException

createDirectory

public void createDirectory(AlluxioURI path,
                            CreateDirectoryPOptions options)
                     throws FileAlreadyExistsException,
                            InvalidPathException,
                            IOException,
                            AlluxioException

Description copied from interface: FileSystem

Creates a directory.

Specified by:

createDirectory in interface FileSystem

Parameters:

path - the path of the directory to create in Alluxio space

options - options to associate with this operation

Throws:

FileAlreadyExistsException - if there is already a file or directory at the given path

InvalidPathException - if the path is invalid

IOException

AlluxioException

createFile

public FileOutStream createFile(AlluxioURI path,
                                CreateFilePOptions options)
                         throws FileAlreadyExistsException,
                                InvalidPathException,
                                IOException,
                                AlluxioException

Description copied from interface: FileSystem

Creates a file.

Specified by:

createFile in interface FileSystem

Parameters:

path - the path of the file to create in Alluxio space

options - options to associate with this operation

Returns:

a FileOutStream which will write data to the newly created file

Throws:

FileAlreadyExistsException - if there is already a file at the given path

InvalidPathException - if the path is invalid

IOException

AlluxioException

delete

public void delete(AlluxioURI path,
                   DeletePOptions options)
            throws DirectoryNotEmptyException,
                   FileDoesNotExistException,
                   IOException,
                   AlluxioException

Description copied from interface: FileSystem

Deletes a file or a directory.

Specified by:

delete in interface FileSystem

Parameters:

path - the path to delete in Alluxio space

options - options to associate with this operation

Throws:

DirectoryNotEmptyException - if recursive is false and the path is a nonempty directory

FileDoesNotExistException - if the given path does not exist

IOException

AlluxioException

exists

public boolean exists(AlluxioURI path,
                      ExistsPOptions options)
               throws IOException,
                      AlluxioException

Description copied from interface: FileSystem

Checks whether a path exists in Alluxio space.

Specified by:

exists in interface FileSystem

Parameters:

path - the path in question

options - options to associate with this operation

Returns:

true if the path exists, false otherwise

Throws:

InvalidPathException - if the path is invalid

IOException

AlluxioException

free

public void free(AlluxioURI path,
                 FreePOptions options)
          throws FileDoesNotExistException,
                 IOException,
                 AlluxioException

Description copied from interface: FileSystem

Evicts any data under the given path from Alluxio space, but does not delete the data from the UFS. The metadata will still be present in Alluxio space after this operation.

Specified by:

free in interface FileSystem

Parameters:

path - the path to free in Alluxio space

options - options to associate with this operation

Throws:

FileDoesNotExistException - if the given path does not exist

IOException

AlluxioException

getBlockLocations

public List<BlockLocationInfo> getBlockLocations(URIStatus status)
                                          throws IOException,
                                                 AlluxioException

Description copied from interface: FileSystem

Builds a list of BlockLocationInfo for the given file.

Specified by:

getBlockLocations in interface FileSystem

Parameters:

status - the path status

Returns:

a list of blocks with the workers whose hosts have the blocks. The blocks may not necessarily be stored in Alluxio. The blocks are returned in the order of their sequences in file.

Throws:

IOException

AlluxioException

getConf

public AlluxioConfiguration getConf()

Specified by:

getConf in interface FileSystem

Returns:

the configuration which the FileSystem is using to connect to Alluxio

getStatus

public URIStatus getStatus(AlluxioURI path,
                           GetStatusPOptions options)
                    throws FileDoesNotExistException,
                           IOException,
                           AlluxioException

Description copied from interface: FileSystem

Gets the URIStatus object that represents the metadata of an Alluxio path.

Specified by:

getStatus in interface FileSystem

Parameters:

path - the path to obtain information about

options - options to associate with this operation

Returns:

the URIStatus of the file

Throws:

FileDoesNotExistException - if the path does not exist

IOException

AlluxioException

listStatus

public List<URIStatus> listStatus(AlluxioURI path,
                                  ListStatusPOptions options)
                           throws FileDoesNotExistException,
                                  IOException,
                                  AlluxioException

Description copied from interface: FileSystem

If the path is a directory, returns the URIStatus of all the direct entries in it. Otherwise returns a list with a single URIStatus element for the file.

Specified by:

listStatus in interface FileSystem

Parameters:

path - the path to list information about

options - options to associate with this operation

Returns:

a list of URIStatuss containing information about the files and directories which are children of the given path

Throws:

FileDoesNotExistException - if the given path does not exist

IOException

AlluxioException

iterateStatus

public void iterateStatus(AlluxioURI path,
                          ListStatusPOptions options,
                          java.util.function.Consumer<? super URIStatus> action)
                   throws FileDoesNotExistException,
                          IOException,
                          AlluxioException

Description copied from interface: FileSystem

Performs a specific action on each URIStatus in the result of FileSystem.listStatus(alluxio.AlluxioURI). This method is preferred when iterating over directories with a large number of files or sub-directories inside. The caller can proceed with partial result without waiting for all result returned.

Specified by:

iterateStatus in interface FileSystem

Parameters:

path - the path to list information about

options - options to associate with this operation

action - action to apply on each URIStatus

Throws:

FileDoesNotExistException - if the given path does not exist

IOException

AlluxioException

listStatusPartial

public ListStatusPartialResult listStatusPartial(AlluxioURI path,
                                                 ListStatusPartialPOptions options)
                                          throws AlluxioException,
                                                 IOException

Description copied from interface: FileSystem

Same as FileSystem.listStatus(AlluxioURI, ListStatusPOptions) except may only return a subset of the results as determined by the options parameter.

Specified by:

listStatusPartial in interface FileSystem

Parameters:

path - the path to list information about

options - options to associate with this operation

Returns:

a list of URIStatuss containing information about the files and directories which are children of the given path

Throws:

FileDoesNotExistException - if the given path does not exist

AlluxioException

IOException

loadMetadata

public void loadMetadata(AlluxioURI path,
                         ListStatusPOptions options)
                  throws FileDoesNotExistException,
                         IOException,
                         AlluxioException

Description copied from interface: FileSystem

Loads metadata about a path in the UFS to Alluxio. No data will be transferred.

Specified by:

loadMetadata in interface FileSystem

Parameters:

path - the path for which to load metadata from UFS

options - options to associate with this operation

Throws:

FileDoesNotExistException - if the given path does not exist

IOException

AlluxioException

mount

public void mount(AlluxioURI alluxioPath,
                  AlluxioURI ufsPath,
                  MountPOptions options)
           throws IOException,
                  AlluxioException

Description copied from interface: FileSystem

Mounts a UFS subtree to the given Alluxio path. The Alluxio path is expected not to exist as the method creates it. If the path already exists, a AlluxioException will be thrown. This method does not transfer any data or metadata from the UFS. It simply establishes the connection between the given Alluxio path and UFS path.

Specified by:

mount in interface FileSystem

Parameters:

alluxioPath - an Alluxio path to mount the data to

ufsPath - a UFS path to mount the data from

options - options to associate with this operation

Throws:

IOException

AlluxioException

updateMount

public void updateMount(AlluxioURI alluxioPath,
                        MountPOptions options)
                 throws IOException,
                        AlluxioException

Description copied from interface: FileSystem

Updates the options for an existing mount point.

Specified by:

updateMount in interface FileSystem

Parameters:

alluxioPath - the Alluxio path of the mount point

options - options for this mount point

Throws:

IOException

AlluxioException

getMountTable

public Map<String,MountPointInfo> getMountTable(boolean checkUfs)
                                         throws IOException,
                                                AlluxioException

Description copied from interface: FileSystem

Lists all mount points and their corresponding under storage addresses.

Specified by:

getMountTable in interface FileSystem

Parameters:

checkUfs - whether to get UFS usage info

Returns:

a map from String to MountPointInfo

Throws:

IOException

AlluxioException

getSyncPathList

public List<SyncPointInfo> getSyncPathList()
                                    throws IOException,
                                           AlluxioException

Description copied from interface: FileSystem

Lists all the actively synced paths.

Specified by:

getSyncPathList in interface FileSystem

Returns:

a list of actively synced paths

Throws:

IOException

AlluxioException

persist

public void persist(AlluxioURI path,
                    ScheduleAsyncPersistencePOptions options)
             throws FileDoesNotExistException,
                    IOException,
                    AlluxioException

Description copied from interface: FileSystem

Schedules the given path to be asynchronously persisted to the under file system. To persist synchronously please see FileSystemUtils.persistAndWait(FileSystem, AlluxioURI, long).

Specified by:

persist in interface FileSystem

Parameters:

path - the uri of the file to persist

options - the options to use when submitting persist the path

Throws:

FileDoesNotExistException

IOException

AlluxioException

openFile

public FileInStream openFile(AlluxioURI path,
                             OpenFilePOptions options)
                      throws FileDoesNotExistException,
                             OpenDirectoryException,
                             FileIncompleteException,
                             IOException,
                             AlluxioException

Description copied from interface: FileSystem

Opens a file for reading.

Specified by:

openFile in interface FileSystem

Parameters:

path - the file to read from

options - options to associate with this operation

Returns:

a FileInStream for the given path

Throws:

FileDoesNotExistException - when path does not exist

OpenDirectoryException - when path is a directory

FileIncompleteException - when path is a file and is not completed yet

IOException

AlluxioException

openFile

public FileInStream openFile(URIStatus status,
                             OpenFilePOptions options)
                      throws FileDoesNotExistException,
                             OpenDirectoryException,
                             FileIncompleteException,
                             IOException,
                             AlluxioException

Description copied from interface: FileSystem

Opens a file for reading.

Specified by:

openFile in interface FileSystem

Parameters:

status - status of the file to read from

options - options to associate with this operation

Returns:

a FileInStream for the given path

Throws:

FileDoesNotExistException - when path does not exist

OpenDirectoryException - when path is a directory

FileIncompleteException - when path is a file and is not completed yet

IOException

AlluxioException

rename

public void rename(AlluxioURI src,
                   AlluxioURI dst,
                   RenamePOptions options)
            throws FileDoesNotExistException,
                   IOException,
                   AlluxioException

Description copied from interface: FileSystem

Renames an existing Alluxio path to another Alluxio path in Alluxio. This operation will be propagated in the underlying storage if the path is persisted.

Specified by:

rename in interface FileSystem

Parameters:

src - the path of the source, this must already exist

dst - the path of the destination, this path should not exist

options - options to associate with this operation

Throws:

FileDoesNotExistException - if the given file does not exist

IOException

AlluxioException

reverseResolve

public AlluxioURI reverseResolve(AlluxioURI ufsUri)
                          throws IOException,
                                 AlluxioException

Description copied from interface: FileSystem

Reverse resolve a ufs uri.

Specified by:

reverseResolve in interface FileSystem

Parameters:

ufsUri - the ufs uri

Returns:

the alluxio path for the ufsUri

Throws:

AlluxioStatusException

IOException

AlluxioException

setAcl

public void setAcl(AlluxioURI path,
                   SetAclAction action,
                   List<AclEntry> entries,
                   SetAclPOptions options)
            throws FileDoesNotExistException,
                   IOException,
                   AlluxioException

Description copied from interface: FileSystem

Sets the ACL for a path.

Specified by:

setAcl in interface FileSystem

Parameters:

path - the path to set the ACL for

action - the set action to perform

entries - the ACL entries

options - options to associate with this operation

Throws:

FileDoesNotExistException - if the given file does not exist

IOException

AlluxioException

setAttribute

public void setAttribute(AlluxioURI path,
                         SetAttributePOptions options)
                  throws FileDoesNotExistException,
                         IOException,
                         AlluxioException

Description copied from interface: FileSystem

Sets any number of a path's attributes, such as TTL and pin status.

Specified by:

setAttribute in interface FileSystem

Parameters:

path - the path to set attributes for

options - options to associate with this operation

Throws:

FileDoesNotExistException - if the given file does not exist

IOException

AlluxioException

startSync

public void startSync(AlluxioURI path)
               throws FileDoesNotExistException,
                      IOException,
                      AlluxioException

Starts the active syncing process on an Alluxio path.

Specified by:

startSync in interface FileSystem

Parameters:

path - the path to sync

Throws:

FileDoesNotExistException

IOException

AlluxioException

stopSync

public void stopSync(AlluxioURI path)
              throws FileDoesNotExistException,
                     IOException,
                     AlluxioException

Stops the active syncing process on an Alluxio path.

Specified by:

stopSync in interface FileSystem

Parameters:

path - the path to stop syncing

Throws:

FileDoesNotExistException

IOException

AlluxioException

unmount

public void unmount(AlluxioURI path,
                    UnmountPOptions options)
             throws IOException,
                    AlluxioException

Description copied from interface: FileSystem

Unmounts a UFS subtree identified by the given Alluxio path. The Alluxio path match a previously mounted path. The contents of the subtree rooted at this path are removed from Alluxio but the corresponding UFS subtree is left untouched.

Specified by:

unmount in interface FileSystem

Parameters:

path - an Alluxio path, this must be a mount point

options - options to associate with this operation

Throws:

IOException

AlluxioException

needsSync

public void needsSync(AlluxioURI path)
               throws IOException,
                      AlluxioException

Description copied from interface: FileSystem

Marks the path in Alluxio as needing sync with the UFS. The next time the path or any of its children are accessed they will be synced with the UFS.

Specified by:

needsSync in interface FileSystem

Parameters:

path - the path needing synchronization

Throws:

IOException

AlluxioException

submitJob

public Optional<String> submitJob(JobRequest jobRequest)

Description copied from interface: FileSystem

Submit a job to scheduler.

Specified by:

submitJob in interface FileSystem

Parameters:

jobRequest - the job request

Returns:

job id if job is submitted, empty if a job with description already exists

stopJob

public boolean stopJob(JobDescription jobDescription)

Description copied from interface: FileSystem

Stop a job in scheduler.

Specified by:

stopJob in interface FileSystem

Parameters:

jobDescription - the job description

Returns:

true if job is stopped, false if we cannot find job

getJobProgress

public String getJobProgress(JobDescription jobDescription,
                             JobProgressReportFormat format,
                             boolean verbose)

Description copied from interface: FileSystem

Get progress of a job.

Specified by:

getJobProgress in interface FileSystem

Parameters:

jobDescription - the job description

format - progress report format

verbose - whether to return verbose report

Returns:

the job progress

checkUri

protected void checkUri(AlluxioURI uri)

Checks an AlluxioURI for scheme and authority information. Warn the user and throw an exception if necessary.

wrapAndThrowAlluxioStatusException

protected void wrapAndThrowAlluxioStatusException(AlluxioStatusException e)
                                           throws AlluxioException,
                                                  IOException

Throws:

AlluxioException

IOException

getFileSystemContext

public FileSystemContext getFileSystemContext()

Returns:

the file system context