alluxio.underfs

Class BaseUnderFileSystem

java.lang.Object

alluxio.underfs.BaseUnderFileSystem

All Implemented Interfaces:

AsyncUfsClient, UnderFileSystem, Closeable, AutoCloseable

Direct Known Subclasses:

ConsistentUnderFileSystem, ObjectUnderFileSystem

@ThreadSafe
public abstract class BaseUnderFileSystem
extends Object
implements UnderFileSystem

A base abstract UnderFileSystem.

Nested Class Summary

Nested classes/interfaces inherited from interface alluxio.underfs.UnderFileSystem

UnderFileSystem.Factory, UnderFileSystem.SpaceType

Field Summary

Fields

Modifier and Type	Field and Description
static Pair<AccessControlList,DefaultAccessControlList>	EMPTY_ACL
protected UnderFileSystemConfiguration	mUfsConf UFS Configuration options.
protected AlluxioURI	mUri The UFS AlluxioURI used to create this BaseUnderFileSystem.

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	BaseUnderFileSystem(AlluxioURI uri, UnderFileSystemConfiguration ufsConf) Constructs an BaseUnderFileSystem.

Method Summary

Modifier and Type	Method and Description
void	close()
OutputStream	create(String path) Creates a file in the under file system with the indicated name.
boolean	deleteDirectory(String path) Deletes a directory from the under file system with the indicated name non-recursively.
boolean	exists(String path) Checks if a file or directory exists in under file system.
Pair<AccessControlList,DefaultAccessControlList>	getAclPair(String path) Gets the ACL and the Default ACL of a file or directory in under file system.
AlluxioConfiguration	getConfiguration() Gets the under file system configuration.
String	getFingerprint(String path) Computes and returns a fingerprint for the path.
UfsMode	getOperationMode(Map<String,UfsMode> physicalUfsState) An UnderFileSystem may be composed of one or more "physical UFS"s.
Fingerprint	getParsedFingerprint(String path) Computes and returns a fingerprint for the path.
Fingerprint	getParsedFingerprint(String path, String contentHash) Same as UnderFileSystem.getParsedFingerprint(String) except, will use the given content hash as the Fingerprint.Tag.CONTENT_HASH field of the fingerprint if non-null.
List<String>	getPhysicalStores() An UnderFileSystem may be composed of one or more "physical UFS"s.
boolean	isObjectStorage()
boolean	isSeekable() Denotes if the under storage supports seeking.
UfsStatus[]	listStatus(String path, ListOptions options) Returns an array of statuses of the files and directories in the directory denoted by this abstract pathname, with options.
Iterator<UfsStatus>	listStatusIterable(String path, ListOptions options, String startAfter, int batchSize) Lists the ufs statuses iteratively.
boolean	mkdirs(String path) Creates the directory named by this abstract pathname.
InputStream	open(String path) Opens an InputStream for a file in under filesystem at the indicated path.
void	performListingAsync(String path, String continuationToken, String startAfter, DescendantType descendantType, boolean checkStatus, java.util.function.Consumer<UfsLoadResult> onComplete, java.util.function.Consumer<Throwable> onError) Lists the ufs statuses for a given path.
AlluxioURI	resolveUri(AlluxioURI ufsBaseUri, String alluxioPath) Returns an AlluxioURI representation for the UnderFileSystem given a base UFS URI, and the Alluxio path from the base.
void	setAclEntries(String path, List<AclEntry> aclEntries) Sets the access control list of a file or directory in under file system.
protected static String	validatePath(String path) Clean the path by creating a URI and turning it back to a string.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface alluxio.underfs.UnderFileSystem

cleanup, connectFromMaster, connectFromWorker, create, createNonexistingFile, createNonexistingFile, deleteDirectory, deleteExistingDirectory, deleteExistingDirectory, deleteExistingFile, deleteFile, getBlockSizeByte, getDirectoryStatus, getExistingDirectoryStatus, getExistingFileStatus, getExistingStatus, getFileLocations, getFileLocations, getFileStatus, getFileStatus, getSpace, getStatus, getStatus, getTemporaryToken, getUnderFSType, isDirectory, isExistingDirectory, isFile, isTemporaryConnectionToken, listStatus, listStatuses, mkdirs, open, openExistingFile, openExistingFile, openPositionRead, refreshTemporaryToken, renameDirectory, renameFile, renameRenamableDirectory, renameRenamableFile, setMode, setOwner, setTemporaryTokenRefreshContext, supportsFlush

Field Detail

EMPTY_ACL

public static final Pair<AccessControlList,DefaultAccessControlList> EMPTY_ACL

mUri

protected final AlluxioURI mUri

The UFS AlluxioURI used to create this BaseUnderFileSystem.

mUfsConf

protected final UnderFileSystemConfiguration mUfsConf

UFS Configuration options.

Constructor Detail

BaseUnderFileSystem

protected BaseUnderFileSystem(AlluxioURI uri,
                              UnderFileSystemConfiguration ufsConf)

Constructs an BaseUnderFileSystem.

Parameters:

uri - the AlluxioURI used to create this ufs

ufsConf - UFS configuration

Method Detail

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException

create

public OutputStream create(String path)
                    throws IOException

Description copied from interface: UnderFileSystem

Creates a file in the under file system with the indicated name. Parents directories will be created recursively.

Specified by:

create in interface UnderFileSystem

Parameters:

path - the file name

Returns:

A OutputStream object

Throws:

IOException

deleteDirectory

public boolean deleteDirectory(String path)
                        throws IOException

Description copied from interface: UnderFileSystem

Deletes a directory from the under file system with the indicated name non-recursively. A non-recursive delete is successful only if the directory is empty.

Specified by:

deleteDirectory in interface UnderFileSystem

Parameters:

path - of the directory to delete

Returns:

true if directory was found and deleted, false otherwise

Throws:

IOException

exists

public boolean exists(String path)
               throws IOException

Description copied from interface: UnderFileSystem

Checks if a file or directory exists in under file system.

Specified by:

exists in interface UnderFileSystem

Parameters:

path - the absolute path

Returns:

true if the path exists, false otherwise

Throws:

IOException

getAclPair

public Pair<AccessControlList,DefaultAccessControlList> getAclPair(String path)
                                                            throws IOException

Description copied from interface: UnderFileSystem

Gets the ACL and the Default ACL of a file or directory in under file system.

Specified by:

getAclPair in interface UnderFileSystem

Parameters:

path - the path to the file or directory

Returns:

the access control list, along with a Default ACL if it is a directory return null if ACL is unsupported or disabled

Throws:

IOException - if ACL is supported and enabled but cannot be retrieved

setAclEntries

public void setAclEntries(String path,
                          List<AclEntry> aclEntries)
                   throws IOException

Description copied from interface: UnderFileSystem

Sets the access control list of a file or directory in under file system. if the ufs does not support acls, this is a noop. This will overwrite the ACL and defaultACL in the UFS.

Specified by:

setAclEntries in interface UnderFileSystem

Parameters:

path - the path to the file or directory

aclEntries - the access control list + default acl represented in a list of acl entries

Throws:

IOException

getConfiguration

public AlluxioConfiguration getConfiguration()

Description copied from interface: UnderFileSystem

Gets the under file system configuration.

Specified by:

getConfiguration in interface UnderFileSystem

Returns:

the configuration

getFingerprint

public String getFingerprint(String path)

Description copied from interface: UnderFileSystem

Computes and returns a fingerprint for the path. The fingerprint is used to determine if two UFS files are identical. The fingerprint must be deterministic, and must not change if a file is only renamed (identical content and permissions). Returns Constants.INVALID_UFS_FINGERPRINT if there is any error.

Specified by:

getFingerprint in interface UnderFileSystem

Parameters:

path - the path to compute the fingerprint for

Returns:

the string representing the fingerprint

getParsedFingerprint

public Fingerprint getParsedFingerprint(String path)

Description copied from interface: UnderFileSystem

Computes and returns a fingerprint for the path. The fingerprint is used to determine if two UFS files are identical. The fingerprint must be deterministic, and must not change if a file is only renamed (identical content and permissions). Returns Fingerprint.INVALID_FINGERPRINT if there is any error. The default implementation relies on UnderFileSystem.getFingerprint(String) and there is one extra parsing. This default implementation is mainly for backward compatibility. Override this for performance.

Specified by:

getParsedFingerprint in interface UnderFileSystem

Parameters:

path - the path to compute the fingerprint for

Returns:

the string representing the fingerprint

getParsedFingerprint

public Fingerprint getParsedFingerprint(String path,
                                        @Nullable
                                        String contentHash)

Description copied from interface: UnderFileSystem

Same as UnderFileSystem.getParsedFingerprint(String) except, will use the given content hash as the Fingerprint.Tag.CONTENT_HASH field of the fingerprint if non-null. This is intended to be used when the file is already in Alluxio and a fingerprint is being created based on that file where the content hash has already been computed.

Specified by:

getParsedFingerprint in interface UnderFileSystem

Parameters:

path - the path to compute the fingerprint for

contentHash - is used as the Fingerprint.Tag.CONTENT_HASH field when creating the fingerprint.

Returns:

the string representing the fingerprint

getOperationMode

public UfsMode getOperationMode(Map<String,UfsMode> physicalUfsState)

Description copied from interface: UnderFileSystem

An UnderFileSystem may be composed of one or more "physical UFS"s. This method is used to determine the operation mode based on the physical UFS operation modes. For example, if this UnderFileSystem is composed of physical UFS hdfs://ns1/ and hdfs://ns2/ with read operations split b/w the two, with physicalUfsState{hdfs://ns1/:NO_ACCESS, hdfs://ns2/:READ_WRITE} this method can return READ_ONLY to allow reads to proceed from hdfs://ns2/.

Specified by:

getOperationMode in interface UnderFileSystem

Parameters:

physicalUfsState - the state of physical UFSs for this UnderFileSystem; keys are expected to be normalized (ending with /)

Returns:

the desired operation mode for this UFS

getPhysicalStores

public List<String> getPhysicalStores()

Description copied from interface: UnderFileSystem

An UnderFileSystem may be composed of one or more "physical UFS"s. This method returns all underlying physical stores; normalized with only scheme and authority.

Specified by:

getPhysicalStores in interface UnderFileSystem

Returns:

physical UFSs this UnderFileSystem is composed of

isObjectStorage

public boolean isObjectStorage()

Specified by:

isObjectStorage in interface UnderFileSystem

Returns:

true if under storage is an object store, false otherwise

isSeekable

public boolean isSeekable()

Description copied from interface: UnderFileSystem

Denotes if the under storage supports seeking. Note, the under file system subclass that returns true for this method should return the input stream extending SeekableUnderFileInputStream in the UnderFileSystem.open(String, OpenOptions) method.

Specified by:

isSeekable in interface UnderFileSystem

Returns:

true if under storage is seekable, false otherwise

listStatusIterable

@Nullable
public Iterator<UfsStatus> listStatusIterable(String path,
                                                        ListOptions options,
                                                        String startAfter,
                                                        int batchSize)
                                                 throws IOException

Description copied from interface: UnderFileSystem

Lists the ufs statuses iteratively.

Specified by:

listStatusIterable in interface UnderFileSystem

Parameters:

path - the abstract pathname to list

options - for list directory

startAfter - the start after token

batchSize - the batch size

Returns:

An iterator of ufs status. Returns null if this abstract pathname does not denote a directory.

Throws:

IOException

performListingAsync

public void performListingAsync(String path,
                                @Nullable
                                String continuationToken,
                                @Nullable
                                String startAfter,
                                DescendantType descendantType,
                                boolean checkStatus,
                                java.util.function.Consumer<UfsLoadResult> onComplete,
                                java.util.function.Consumer<Throwable> onError)

Description copied from interface: AsyncUfsClient

Lists the ufs statuses for a given path. The UfsStatus.getName() function for the returned values should include the full path of each item from the UFS root (not including the bucket name for object stores). It differs from a traditional listing in that if the input variable checkStatus is true, the UfsStatus for the base path should be included at the start of the results. The function should return immediately, and perform the operation asynchronously.

Specified by:

performListingAsync in interface AsyncUfsClient

Parameters:

path - the path in ufs

continuationToken - the continuation token

startAfter - the start after string where the loading starts from

descendantType - the load descendant type (NONE/ONE/ALL)

checkStatus - if true the call will perform a GetStatus on the path to see if an object exists, which should be returned as part of the result

onComplete - the callback when the load is complete

onError - the callback when the load encountered an error

listStatus

@Nullable
public UfsStatus[] listStatus(String path,
                                        ListOptions options)
                                 throws IOException

Description copied from interface: UnderFileSystem

Returns an array of statuses of the files and directories in the directory denoted by this abstract pathname, with options.

If this abstract pathname does not denote a directory, then this method returns null. Otherwise an array of statuses is returned, one for each file or directory. Names denoting the directory itself and the directory's parent directory are not included in the result. Each string is a path relative to the given directory.

There is no guarantee that the name strings in the resulting array will appear in any specific order; they are not, in particular, guaranteed to appear in alphabetical order.

Specified by:

listStatus in interface UnderFileSystem

Parameters:

path - the abstract pathname to list

options - for list directory

Returns:

An array of statuses naming the files and directories in the directory denoted by this abstract pathname. The array will be empty if the directory is empty. Returns null if this abstract pathname does not denote a directory.

Throws:

IOException

open

public InputStream open(String path)
                 throws IOException

Description copied from interface: UnderFileSystem

Opens an InputStream for a file in under filesystem at the indicated path.

Specified by:

open in interface UnderFileSystem

Parameters:

path - the file name

Returns:

The InputStream object

Throws:

IOException

mkdirs

public boolean mkdirs(String path)
               throws IOException

Description copied from interface: UnderFileSystem

Creates the directory named by this abstract pathname. If the folder already exists, the method returns false. The method creates any necessary but nonexistent parent directories.

Specified by:

mkdirs in interface UnderFileSystem

Parameters:

path - the folder to create

Returns:

true if and only if the directory was created; false otherwise

Throws:

IOException

resolveUri

public AlluxioURI resolveUri(AlluxioURI ufsBaseUri,
                             String alluxioPath)

Description copied from interface: UnderFileSystem

Returns an AlluxioURI representation for the UnderFileSystem given a base UFS URI, and the Alluxio path from the base. The default implementation simply concatenates the path to the base URI. This should be overridden if a subclass needs alternate functionality.

Specified by:

resolveUri in interface UnderFileSystem

Parameters:

ufsBaseUri - the base AlluxioURI in the ufs

alluxioPath - the path in Alluxio from the given base

Returns:

the UFS AlluxioURI representing the Alluxio path

validatePath

protected static String validatePath(String path)

Clean the path by creating a URI and turning it back to a string.

Parameters:

path - the path to validate

Returns:

validated path