alluxio.underfs.hdfs

Class HdfsUnderFileSystem

java.lang.Object

alluxio.underfs.BaseUnderFileSystem

alluxio.underfs.ConsistentUnderFileSystem

alluxio.underfs.hdfs.HdfsUnderFileSystem

All Implemented Interfaces:

AsyncUfsClient, AtomicFileOutputStreamCallback, UnderFileSystem, Closeable, AutoCloseable

Direct Known Subclasses:

AbfsUnderFileSystem, AdlUnderFileSystem, CephfsHadoopUnderFileSystem, CosnUnderFileSystem, OzoneUnderFileSystem, WasbUnderFileSystem

@ThreadSafe
public class HdfsUnderFileSystem
extends ConsistentUnderFileSystem
implements AtomicFileOutputStreamCallback

HDFS UnderFileSystem implementation.

Nested Class Summary

Nested classes/interfaces inherited from interface alluxio.underfs.UnderFileSystem

UnderFileSystem.Factory, UnderFileSystem.SpaceType

Field Summary

Fields

Modifier and Type	Field and Description
protected static String	CHECKSUM_COMBINE_MODE
protected static String	HADOOP_AUTH_METHOD
protected static String	HDFS_ACL_PROVIDER_CLASS Name of the class for the HDFS Acl provider.
protected static String	HDFS_EC_CODEC_REGISTRY_CLASS Name of the class for the HDFS EC Codec Registry.
protected static String	HDFS_EC_MIN_VERSION The minimum HDFS production version required for EC.
protected static String	HDFS_USER
protected static String	JAVAX_WS_RS_CORE_MEDIA_TYPE
protected static String	KRB_KEYTAB_LOGIN_AUTO_RENEW
protected static String	KRB5_CONF_FILE
protected static int	MAX_TRY
protected HdfsAclProvider	mHdfsAclProvider

Fields inherited from class alluxio.underfs.BaseUnderFileSystem

EMPTY_ACL, mUfsConf, mUri

Constructor Summary

Constructors

Modifier	Constructor and Description
	HdfsUnderFileSystem(AlluxioURI ufsUri, UnderFileSystemConfiguration conf, org.apache.hadoop.conf.Configuration hdfsConf) Constructs a new HDFS UnderFileSystem.
protected	HdfsUnderFileSystem(AlluxioURI ufsUri, UnderFileSystemConfiguration conf, org.apache.hadoop.conf.Configuration hdfsConf, boolean useLoadingCache)

Method Summary

Modifier and Type	Method and Description
void	cleanup() Cleans up the under file system.
void	close()
void	connectFromMaster(String host) Takes any necessary actions required to establish a connection to the under file system from the given master host e.g.
void	connectFromWorker(String host) Takes any necessary actions required to establish a connection to the under file system from the given worker host e.g.
OutputStream	create(String path, CreateOptions options) Creates a file in the under file system with the specified CreateOptions.
static org.apache.hadoop.conf.Configuration	createConfiguration(UnderFileSystemConfiguration conf) Prepares the Hadoop configuration necessary to successfully obtain a FileSystem instance that can access the provided path.
OutputStream	createDirect(String path, CreateOptions options) Creates a file in the under file system with the specified CreateOptions.
static HdfsUnderFileSystem	createInstance(AlluxioURI ufsUri, UnderFileSystemConfiguration conf) Factory method to constructs a new HDFS UnderFileSystem instance.
boolean	deleteDirectory(String path, DeleteOptions options) Deletes a directory from the under file system with the indicated name.
boolean	deleteFile(String path) Deletes a file from the under file system with the indicated name.
boolean	exists(String path) Checks if a file or directory exists in under file system.
Pair<AccessControlList,DefaultAccessControlList>	getAclPair(String path) Gets the ACL and the Default ACL of a file or directory in under file system.
long	getBlockSizeByte(String path) Gets the block size of a file in under file system, in bytes.
UfsDirectoryStatus	getDirectoryStatus(String path) Gets the directory status.
List<String>	getFileLocations(String path) Gets the list of locations of the indicated path.
List<String>	getFileLocations(String path, FileLocationOptions options) Gets the list of locations of the indicated path given options.
UfsFileStatus	getFileStatus(String path, GetStatusOptions options) Gets the file status.
org.apache.hadoop.fs.FileSystem	getFs()
long	getSpace(String path, UnderFileSystem.SpaceType type) Queries the under file system about the space of the indicated path (e.g., space left, space used and etc).
UfsStatus	getStatus(String path, GetStatusOptions options) Gets the file or directory status.
String	getUnderFSType() Returns the name of the under filesystem implementation.
boolean	isDirectory(String path) Checks if a directory exists in under file system.
boolean	isFile(String path) Checks if a file exists in under file system.
boolean	isSeekable() Denotes if the under storage supports seeking.
UfsStatus[]	listStatus(String path) Returns an array of statuses of the files and directories in the directory denoted by this abstract pathname.
boolean	mkdirs(String path, MkdirsOptions options) Creates the directory named by this abstract pathname, with specified MkdirsOptions.
InputStream	open(String path, OpenOptions options) Opens an InputStream for a file in under filesystem at the indicated path.
HdfsPositionedUnderFileInputStream	openPositionRead(String path, long fileLength) Opens a file for position read.
boolean	renameDirectory(String src, String dst) Renames a directory from src to dst in under file system.
boolean	renameFile(String src, String dst) Renames a file from src to dst in under file system.
void	setAclEntries(String path, List<AclEntry> aclEntries) Sets the access control list of a file or directory in under file system.
void	setMode(String path, short mode) Changes posix file mode.
void	setOwner(String path, String user, String group) Sets the user and group of the given path.
boolean	supportsFlush() Whether this type of UFS supports flush.

Methods inherited from class alluxio.underfs.ConsistentUnderFileSystem

createNonexistingFile, createNonexistingFile, deleteExistingDirectory, deleteExistingDirectory, deleteExistingFile, getExistingDirectoryStatus, getExistingFileStatus, getExistingStatus, isExistingDirectory, openExistingFile, openExistingFile, renameRenamableDirectory, renameRenamableFile

Methods inherited from class alluxio.underfs.BaseUnderFileSystem

create, deleteDirectory, getConfiguration, getFingerprint, getOperationMode, getParsedFingerprint, getParsedFingerprint, getPhysicalStores, isObjectStorage, listStatus, listStatusIterable, mkdirs, open, performListingAsync, resolveUri, validatePath

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface alluxio.underfs.UnderFileSystem

create, createNonexistingFile, createNonexistingFile, deleteDirectory, deleteExistingDirectory, deleteExistingDirectory, deleteExistingFile, getConfiguration, getExistingDirectoryStatus, getExistingFileStatus, getExistingStatus, getFileStatus, getFingerprint, getOperationMode, getParsedFingerprint, getParsedFingerprint, getPhysicalStores, getStatus, getTemporaryToken, isExistingDirectory, isObjectStorage, isTemporaryConnectionToken, listStatus, listStatuses, listStatusIterable, mkdirs, open, openExistingFile, openExistingFile, refreshTemporaryToken, renameRenamableDirectory, renameRenamableFile, resolveUri, setTemporaryTokenRefreshContext

Methods inherited from interface alluxio.underfs.AsyncUfsClient

performListingAsync

Field Detail

MAX_TRY

protected static final int MAX_TRY

See Also:

Constant Field Values

HDFS_USER

protected static final String HDFS_USER

See Also:

Constant Field Values

HDFS_ACL_PROVIDER_CLASS

protected static final String HDFS_ACL_PROVIDER_CLASS

Name of the class for the HDFS Acl provider.

See Also:

Constant Field Values

HDFS_EC_MIN_VERSION

protected static final String HDFS_EC_MIN_VERSION

The minimum HDFS production version required for EC.

See Also:

Constant Field Values

HDFS_EC_CODEC_REGISTRY_CLASS

protected static final String HDFS_EC_CODEC_REGISTRY_CLASS

Name of the class for the HDFS EC Codec Registry.

See Also:

Constant Field Values

JAVAX_WS_RS_CORE_MEDIA_TYPE

protected static final String JAVAX_WS_RS_CORE_MEDIA_TYPE

See Also:

Constant Field Values

HADOOP_AUTH_METHOD

protected static final String HADOOP_AUTH_METHOD

See Also:

Constant Field Values

KRB5_CONF_FILE

protected static final String KRB5_CONF_FILE

See Also:

Constant Field Values

KRB_KEYTAB_LOGIN_AUTO_RENEW

protected static final String KRB_KEYTAB_LOGIN_AUTO_RENEW

See Also:

Constant Field Values

CHECKSUM_COMBINE_MODE

protected static final String CHECKSUM_COMBINE_MODE

See Also:

Constant Field Values

mHdfsAclProvider

protected final HdfsAclProvider mHdfsAclProvider

Constructor Detail

HdfsUnderFileSystem

public HdfsUnderFileSystem(AlluxioURI ufsUri,
                           UnderFileSystemConfiguration conf,
                           org.apache.hadoop.conf.Configuration hdfsConf)

Constructs a new HDFS UnderFileSystem.

Parameters:

ufsUri - the AlluxioURI for this UFS

conf - the configuration for this UFS

hdfsConf - the configuration for HDFS

HdfsUnderFileSystem

protected HdfsUnderFileSystem(AlluxioURI ufsUri,
                              UnderFileSystemConfiguration conf,
                              org.apache.hadoop.conf.Configuration hdfsConf,
                              boolean useLoadingCache)

Method Detail

createInstance

public static HdfsUnderFileSystem createInstance(AlluxioURI ufsUri,
                                                 UnderFileSystemConfiguration conf)

Factory method to constructs a new HDFS UnderFileSystem instance.

Parameters:

ufsUri - the AlluxioURI for this UFS

conf - the configuration for Hadoop

Returns:

a new HDFS UnderFileSystem instance

getUnderFSType

public String getUnderFSType()

Description copied from interface: UnderFileSystem

Returns the name of the under filesystem implementation. The name should be lowercase and not include any spaces, e.g. "hdfs", "s3".

Specified by:

getUnderFSType in interface UnderFileSystem

Returns:

name of the under filesystem implementation

createConfiguration

public static org.apache.hadoop.conf.Configuration createConfiguration(UnderFileSystemConfiguration conf)

Prepares the Hadoop configuration necessary to successfully obtain a FileSystem instance that can access the provided path.

Derived implementations that work with specialised Hadoop FileSystem API compatible implementations can override this method to add implementation specific configuration necessary for obtaining a usable FileSystem instance.

Parameters:

conf - the configuration for this UFS

Returns:

the configuration for HDFS

cleanup

public void cleanup()

Description copied from interface: UnderFileSystem

Cleans up the under file system. If any data or files are created and not completed/aborted correctly in normal ways, they should be cleaned in this method.

Specified by:

cleanup in interface UnderFileSystem

close

public void close()

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Overrides:

close in class BaseUnderFileSystem

create

public OutputStream create(String path,
                           CreateOptions options)
                    throws IOException

Description copied from interface: UnderFileSystem

Creates a file in the under file system with the specified CreateOptions. Implementations should make sure that the path under creation appears in listings only after a successful close and that contents are written in its entirety or not at all.

Specified by:

create in interface UnderFileSystem

Parameters:

path - the file name

options - the options for create

Returns:

A OutputStream object

Throws:

IOException

createDirect

public OutputStream createDirect(String path,
                                 CreateOptions options)
                          throws IOException

Description copied from interface: AtomicFileOutputStreamCallback

Creates a file in the under file system with the specified CreateOptions. This stream writes directly to the underlying storage without any atomicity guarantees.

Specified by:

createDirect in interface AtomicFileOutputStreamCallback

Parameters:

path - the file name

options - the options for create

Returns:

A OutputStream object

Throws:

IOException

deleteDirectory

public boolean deleteDirectory(String path,
                               DeleteOptions options)
                        throws IOException

Description copied from interface: UnderFileSystem

Deletes a directory from the under file system with the indicated name.

Specified by:

deleteDirectory in interface UnderFileSystem

Parameters:

path - of the directory to delete

options - for directory delete semantics

Returns:

true if directory was found and deleted, false otherwise

Throws:

IOException

deleteFile

public boolean deleteFile(String path)
                   throws IOException

Description copied from interface: UnderFileSystem

Deletes a file from the under file system with the indicated name.

Specified by:

deleteFile in interface UnderFileSystem

Parameters:

path - of the file to delete

Returns:

true if file was found and deleted, false otherwise

Throws:

IOException

exists

public boolean exists(String path)
               throws IOException

Description copied from interface: UnderFileSystem

Checks if a file or directory exists in under file system.

Specified by:

exists in interface UnderFileSystem

Overrides:

exists in class BaseUnderFileSystem

Parameters:

path - the absolute path

Returns:

true if the path exists, false otherwise

Throws:

IOException

getAclPair

public Pair<AccessControlList,DefaultAccessControlList> getAclPair(String path)
                                                            throws IOException

Description copied from interface: UnderFileSystem

Gets the ACL and the Default ACL of a file or directory in under file system.

Specified by:

getAclPair in interface UnderFileSystem

Overrides:

getAclPair in class BaseUnderFileSystem

Parameters:

path - the path to the file or directory

Returns:

the access control list, along with a Default ACL if it is a directory return null if ACL is unsupported or disabled

Throws:

IOException - if ACL is supported and enabled but cannot be retrieved

setAclEntries

public void setAclEntries(String path,
                          List<AclEntry> aclEntries)
                   throws IOException

Description copied from interface: UnderFileSystem

Sets the access control list of a file or directory in under file system. if the ufs does not support acls, this is a noop. This will overwrite the ACL and defaultACL in the UFS.

Specified by:

setAclEntries in interface UnderFileSystem

Overrides:

setAclEntries in class BaseUnderFileSystem

Parameters:

path - the path to the file or directory

aclEntries - the access control list + default acl represented in a list of acl entries

Throws:

IOException

getBlockSizeByte

public long getBlockSizeByte(String path)
                      throws IOException

Description copied from interface: UnderFileSystem

Gets the block size of a file in under file system, in bytes.

Specified by:

getBlockSizeByte in interface UnderFileSystem

Parameters:

path - the file name

Returns:

the block size in bytes

Throws:

IOException

getDirectoryStatus

public UfsDirectoryStatus getDirectoryStatus(String path)
                                      throws IOException

Description copied from interface: UnderFileSystem

Gets the directory status. The caller must already know the path is a directory. This method will throw an exception if the path exists, but is a file.

Specified by:

getDirectoryStatus in interface UnderFileSystem

Parameters:

path - the path to the directory

Returns:

the directory status

Throws:

FileNotFoundException - when the path does not exist

IOException

getFileLocations

public List<String> getFileLocations(String path)
                              throws IOException

Description copied from interface: UnderFileSystem

Gets the list of locations of the indicated path.

Specified by:

getFileLocations in interface UnderFileSystem

Parameters:

path - the file name

Returns:

The list of locations

Throws:

IOException

getFileLocations

@Nullable
public List<String> getFileLocations(String path,
                                               FileLocationOptions options)
                                        throws IOException

Description copied from interface: UnderFileSystem

Gets the list of locations of the indicated path given options.

Specified by:

getFileLocations in interface UnderFileSystem

Parameters:

path - the file name

options - method options

Returns:

The list of locations

Throws:

IOException

getFileStatus

public UfsFileStatus getFileStatus(String path,
                                   GetStatusOptions options)
                            throws IOException

Description copied from interface: UnderFileSystem

Gets the file status. The caller must already know the path is a file. This method will throw an exception if the path exists, but is a directory.

Specified by:

getFileStatus in interface UnderFileSystem

Parameters:

path - the path to the file

options - method options

Returns:

the file status

Throws:

FileNotFoundException - when the path does not exist

IOException

getSpace

public long getSpace(String path,
                     UnderFileSystem.SpaceType type)
              throws IOException

Description copied from interface: UnderFileSystem

Queries the under file system about the space of the indicated path (e.g., space left, space used and etc).

Specified by:

getSpace in interface UnderFileSystem

Parameters:

path - the path to query

type - the type of queries

Returns:

The space in bytes

Throws:

IOException

getStatus

public UfsStatus getStatus(String path,
                           GetStatusOptions options)
                    throws IOException

Description copied from interface: UnderFileSystem

Gets the file or directory status. The caller does not need to know if the path is a file or directory. This method will determine the path type, and will return the appropriate status.

Specified by:

getStatus in interface UnderFileSystem

Parameters:

path - the path to get the status

options - method options

Returns:

the file or directory status

Throws:

FileNotFoundException - when the path does not exist

IOException

isDirectory

public boolean isDirectory(String path)
                    throws IOException

Description copied from interface: UnderFileSystem

Checks if a directory exists in under file system.

Specified by:

isDirectory in interface UnderFileSystem

Parameters:

path - the absolute directory path

Returns:

true if the path exists and is a directory, false otherwise

Throws:

IOException

isFile

public boolean isFile(String path)
               throws IOException

Description copied from interface: UnderFileSystem

Checks if a file exists in under file system.

Specified by:

isFile in interface UnderFileSystem

Parameters:

path - the absolute file path

Returns:

true if the path exists and is a file, false otherwise

Throws:

IOException

listStatus

@Nullable
public UfsStatus[] listStatus(String path)
                                 throws IOException

Description copied from interface: UnderFileSystem

Returns an array of statuses of the files and directories in the directory denoted by this abstract pathname.

If this abstract pathname does not denote a directory, then this method returns null. Otherwise an array of statuses is returned, one for each file or directory in the directory. Names denoting the directory itself and the directory's parent directory are not included in the result. Each string is a file name rather than a complete path.

There is no guarantee that the name strings in the resulting array will appear in any specific order; they are not, in particular, guaranteed to appear in alphabetical order.

Specified by:

listStatus in interface UnderFileSystem

Parameters:

path - the abstract pathname to list

Returns:

An array with the statuses of the files and directories in the directory denoted by this abstract pathname. The array will be empty if the directory is empty. Returns null if this abstract pathname does not denote a directory.

Throws:

IOException

connectFromMaster

public void connectFromMaster(String host)
                       throws IOException

Description copied from interface: UnderFileSystem

Takes any necessary actions required to establish a connection to the under file system from the given master host e.g. logging in

Depending on the implementation this may be a no-op

Specified by:

connectFromMaster in interface UnderFileSystem

Parameters:

host - the host that wants to connect to the under file system

Throws:

IOException

connectFromWorker

public void connectFromWorker(String host)
                       throws IOException

Description copied from interface: UnderFileSystem

Takes any necessary actions required to establish a connection to the under file system from the given worker host e.g. logging in

Depending on the implementation this may be a no-op

Specified by:

connectFromWorker in interface UnderFileSystem

Parameters:

host - the host that wants to connect to the under file system

Throws:

IOException

mkdirs

public boolean mkdirs(String path,
                      MkdirsOptions options)
               throws IOException

Description copied from interface: UnderFileSystem

Creates the directory named by this abstract pathname, with specified MkdirsOptions. If the folder already exists, the method returns false.

Specified by:

mkdirs in interface UnderFileSystem

Parameters:

path - the folder to create

options - the options for mkdirs

Returns:

true if and only if the directory was created; false otherwise

Throws:

IOException

open

public InputStream open(String path,
                        OpenOptions options)
                 throws IOException

Description copied from interface: UnderFileSystem

Opens an InputStream for a file in under filesystem at the indicated path.

Specified by:

open in interface UnderFileSystem

Parameters:

path - the file name

options - to open input stream

Returns:

The InputStream object

Throws:

IOException

openPositionRead

public HdfsPositionedUnderFileInputStream openPositionRead(String path,
                                                           long fileLength)

Description copied from interface: UnderFileSystem

Opens a file for position read.

Specified by:

openPositionRead in interface UnderFileSystem

Parameters:

path - the path to read

fileLength - the file length

Returns:

the position reader

renameDirectory

public boolean renameDirectory(String src,
                               String dst)
                        throws IOException

Description copied from interface: UnderFileSystem

Renames a directory from src to dst in under file system.

Specified by:

renameDirectory in interface UnderFileSystem

Parameters:

src - the source directory path

dst - the destination directory path

Returns:

true if succeed, false otherwise

Throws:

IOException

renameFile

public boolean renameFile(String src,
                          String dst)
                   throws IOException

Description copied from interface: UnderFileSystem

Renames a file from src to dst in under file system.

Specified by:

renameFile in interface UnderFileSystem

Parameters:

src - the source file path

dst - the destination file path

Returns:

true if succeed, false otherwise

Throws:

IOException

setOwner

public void setOwner(String path,
                     String user,
                     String group)
              throws IOException

Description copied from interface: UnderFileSystem

Sets the user and group of the given path. An empty implementation should be provided if unsupported.

Specified by:

setOwner in interface UnderFileSystem

Parameters:

path - the path of the file

user - the new owner to set, unchanged if null

group - the new group to set, unchanged if null

Throws:

IOException

setMode

public void setMode(String path,
                    short mode)
             throws IOException

Description copied from interface: UnderFileSystem

Changes posix file mode.

Specified by:

setMode in interface UnderFileSystem

Parameters:

path - the path of the file

mode - the mode to set in short format, e.g. 0777

Throws:

IOException

supportsFlush

public boolean supportsFlush()

Description copied from interface: UnderFileSystem

Whether this type of UFS supports flush.

Specified by:

supportsFlush in interface UnderFileSystem

Returns:

true if this type of UFS supports flush, false otherwise

isSeekable

public boolean isSeekable()

Description copied from interface: UnderFileSystem

Denotes if the under storage supports seeking. Note, the under file system subclass that returns true for this method should return the input stream extending SeekableUnderFileInputStream in the UnderFileSystem.open(String, OpenOptions) method.

Specified by:

isSeekable in interface UnderFileSystem

Overrides:

isSeekable in class BaseUnderFileSystem

Returns:

true if under storage is seekable, false otherwise

getFs

public org.apache.hadoop.fs.FileSystem getFs()
                                      throws IOException

Returns:

the underlying HDFS FileSystem object

Throws:

IOException