alluxio.hadoop

Class FileSystem

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.fs.FileSystem

alluxio.hadoop.FileSystem

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.conf.Configurable

@NotThreadSafe
public final class FileSystem
extends org.apache.hadoop.fs.FileSystem

An Alluxio client API compatible with Apache Hadoop FileSystem interface. Any program working with Hadoop HDFS can work with Alluxio transparently. Note that the performance of using this API may not be as efficient as the performance of using the Alluxio native API defined in FileSystem, which this API is built on top of.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.fs.FileSystem

org.apache.hadoop.fs.FileSystem.Statistics

Field Summary

Fields

Modifier and Type	Field and Description
static String	FIRST_COM_PATH
protected FileSystem	mFileSystem

Fields inherited from class org.apache.hadoop.fs.FileSystem

DEFAULT_FS, FS_DEFAULT_NAME_KEY, SHUTDOWN_HOOK_PRIORITY, statistics

Constructor Summary

Constructors

Constructor and Description
FileSystem() Constructs a new FileSystem.
FileSystem(FileSystem fileSystem) Constructs a new FileSystem instance with a specified FileSystem handler for tests.

Method Summary

Modifier and Type	Method and Description
org.apache.hadoop.fs.FSDataOutputStream	append(org.apache.hadoop.fs.Path path, int bufferSize, org.apache.hadoop.util.Progressable progress)
void	close()
org.apache.hadoop.fs.FSDataOutputStream	create(org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.permission.FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress) Attempts to create a file.
org.apache.hadoop.fs.FSDataOutputStream	createNonRecursive(org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.permission.FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress) Deprecated. API only for 0.20-append
boolean	delete(org.apache.hadoop.fs.Path path) Deprecated. Use delete(Path, boolean) instead.
boolean	delete(org.apache.hadoop.fs.Path path, boolean recursive) Attempts to delete the file or directory with the specified path.
protected AlluxioURI	getAlluxioPath(org.apache.hadoop.fs.Path path) Used to convert hadoop path to Alluxio path.
protected Map<String,Object>	getConfigurationFromUri(URI uri) Gets the connection configuration from the input uri.
long	getDefaultBlockSize()
short	getDefaultReplication()
org.apache.hadoop.fs.BlockLocation[]	getFileBlockLocations(org.apache.hadoop.fs.FileStatus file, long start, long len)
org.apache.hadoop.fs.FileStatus	getFileStatus(org.apache.hadoop.fs.Path path) If the file does not exist in Alluxio, query it from HDFS.
protected org.apache.hadoop.fs.Path	getFsPath(String fsUriHeader, URIStatus fileStatus) Used to get FS native path from Alluxio file status.
protected String	getFsScheme(URI fsUri) Used to get FS scheme.
String	getScheme() Gets the URI scheme that maps to the FileSystem.
URI	getUri()
org.apache.hadoop.fs.Path	getWorkingDirectory()
void	initialize(URI uri, org.apache.hadoop.conf.Configuration conf) Sets up a lazy connection to Alluxio through mFileSystem.
void	initialize(URI uri, org.apache.hadoop.conf.Configuration conf, AlluxioConfiguration alluxioConfiguration) Initialize the FileSystem.
protected boolean	isZookeeperMode()
org.apache.hadoop.fs.FileStatus[]	listStatus(org.apache.hadoop.fs.Path path)
boolean	mkdirs(org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.permission.FsPermission permission) Attempts to create a folder with the specified path.
org.apache.hadoop.fs.FSDataInputStream	open(org.apache.hadoop.fs.Path path, int bufferSize) Attempts to open the specified file for reading.
boolean	rename(org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst)
void	setOwner(org.apache.hadoop.fs.Path path, String username, String groupname) Changes owner or group of a path (i.e.
void	setPermission(org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.permission.FsPermission permission) Changes permission of a path.
boolean	setReplication(org.apache.hadoop.fs.Path path, short replication)
void	setWorkingDirectory(org.apache.hadoop.fs.Path path)
protected void	validateFsUri(URI fsUri) Validates given FS base URI for scheme and authority.

Methods inherited from class org.apache.hadoop.fs.FileSystem

access, addDelegationTokens, append, append, areSymlinksEnabled, cancelDeleteOnExit, canonicalizeUri, checkPath, clearStatistics, closeAll, closeAllForUGI, completeLocalOutput, concat, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyToLocalFile, copyToLocalFile, copyToLocalFile, create, create, create, create, create, create, create, create, create, create, create, create, createNewFile, createNonRecursive, createNonRecursive, createSnapshot, createSnapshot, createSymlink, deleteOnExit, deleteSnapshot, enableSymlinks, exists, fixRelativePart, get, get, get, getAclStatus, getAllStatistics, getBlockSize, getCanonicalServiceName, getCanonicalUri, getChildFileSystems, getContentSummary, getDefaultBlockSize, getDefaultPort, getDefaultReplication, getDefaultUri, getDelegationToken, getFileBlockLocations, getFileChecksum, getFileChecksum, getFileLinkStatus, getFileSystemClass, getFSofPath, getHomeDirectory, getInitialWorkingDirectory, getLength, getLinkTarget, getLocal, getName, getNamed, getReplication, getServerDefaults, getServerDefaults, getStatistics, getStatistics, getStatus, getStatus, getUsed, getXAttr, getXAttrs, getXAttrs, globStatus, globStatus, isDirectory, isFile, listCorruptFileBlocks, listFiles, listLocatedStatus, listLocatedStatus, listStatus, listStatus, listStatus, listStatusIterator, listXAttrs, makeQualified, mkdirs, mkdirs, modifyAclEntries, moveFromLocalFile, moveFromLocalFile, moveToLocalFile, newInstance, newInstance, newInstance, newInstanceLocal, open, primitiveCreate, primitiveMkdir, primitiveMkdir, printStatistics, processDeleteOnExit, removeAcl, removeAclEntries, removeDefaultAcl, removeXAttr, rename, renameSnapshot, resolveLink, resolvePath, setAcl, setDefaultUri, setDefaultUri, setTimes, setVerifyChecksum, setWriteChecksum, setXAttr, setXAttr, startLocalOutput, supportsSymlinks, truncate

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

FIRST_COM_PATH

public static final String FIRST_COM_PATH

See Also:

Constant Field Values

mFileSystem

protected FileSystem mFileSystem

Constructor Detail

FileSystem

public FileSystem()

Constructs a new FileSystem.

FileSystem

public FileSystem(FileSystem fileSystem)

Constructs a new FileSystem instance with a specified FileSystem handler for tests.

Parameters:

fileSystem - handler to file system

Method Detail

getScheme

public String getScheme()

Gets the URI scheme that maps to the FileSystem. This was introduced in Hadoop 2.x as a means to make loading new FileSystem s simpler.

Returns:

scheme hadoop should map to

See Also:

FileSystem.createFileSystem(java.net.URI, org.apache.hadoop.conf.Configuration)

isZookeeperMode

protected boolean isZookeeperMode()

Returns:

true if zookeeper should be used

getConfigurationFromUri

protected Map<String,Object> getConfigurationFromUri(URI uri)

Gets the connection configuration from the input uri.

Parameters:

uri - a Alluxio Uri that may contain connection configuration

validateFsUri

protected void validateFsUri(URI fsUri)
                      throws IOException,
                             IllegalArgumentException

Validates given FS base URI for scheme and authority.

Parameters:

fsUri - FS Uri

Throws:

IOException

IllegalArgumentException

getFsScheme

protected String getFsScheme(URI fsUri)

Used to get FS scheme.

Parameters:

fsUri - file system base URI

Returns:

file system scheme

getAlluxioPath

protected AlluxioURI getAlluxioPath(org.apache.hadoop.fs.Path path)

Used to convert hadoop path to Alluxio path.

Parameters:

path - the input path

Returns:

the Alluxio path

getFsPath

protected org.apache.hadoop.fs.Path getFsPath(String fsUriHeader,
                                              URIStatus fileStatus)

Used to get FS native path from Alluxio file status.

Parameters:

fsUriHeader - FS URI header -> "scheme://authority"

fileStatus - Alluxio file status

Returns:

FS native path

append

public org.apache.hadoop.fs.FSDataOutputStream append(org.apache.hadoop.fs.Path path,
                                                      int bufferSize,
                                                      org.apache.hadoop.util.Progressable progress)
                                               throws IOException

Specified by:

append in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Overrides:

close in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

create

public org.apache.hadoop.fs.FSDataOutputStream create(org.apache.hadoop.fs.Path path,
                                                      org.apache.hadoop.fs.permission.FsPermission permission,
                                                      boolean overwrite,
                                                      int bufferSize,
                                                      short replication,
                                                      long blockSize,
                                                      org.apache.hadoop.util.Progressable progress)
                                               throws IOException

Attempts to create a file. Overwrite will not succeed if the path exists and is a folder.

Specified by:

create in class org.apache.hadoop.fs.FileSystem

Parameters:

path - path to create

permission - permissions of the created file/folder

overwrite - overwrite if file exists

bufferSize - the size in bytes of the buffer to be used

replication - under filesystem replication factor, this is ignored

blockSize - block size in bytes

progress - queryable progress

Returns:

an FSDataOutputStream created at the indicated path of a file

Throws:

IOException

createNonRecursive

@Deprecated
public org.apache.hadoop.fs.FSDataOutputStream createNonRecursive(org.apache.hadoop.fs.Path path,
                                                                              org.apache.hadoop.fs.permission.FsPermission permission,
                                                                              boolean overwrite,
                                                                              int bufferSize,
                                                                              short replication,
                                                                              long blockSize,
                                                                              org.apache.hadoop.util.Progressable progress)
                                                                       throws IOException

Deprecated. API only for 0.20-append

Opens an FSDataOutputStream at the indicated Path with write-progress reporting. Same as FileSystem.create(Path, boolean, int, short, long, Progressable), except fails if parent directory doesn't already exist. TODO(hy): We need to refactor this method after having a new internal API support (ALLUXIO-46).

Overrides:

createNonRecursive in class org.apache.hadoop.fs.FileSystem

Parameters:

path - the file name to open

overwrite - if a file with this name already exists, then if true, the file will be overwritten, and if false an error will be thrown.

bufferSize - the size of the buffer to be used

replication - required block replication for the file

blockSize - the size in bytes of the buffer to be used

progress - queryable progress

Throws:

IOException

See Also:

setPermission(Path, FsPermission)

delete

@Deprecated
public boolean delete(org.apache.hadoop.fs.Path path)
                           throws IOException

Deprecated. Use delete(Path, boolean) instead.

Attempts to delete the file or directory with the specified path.

Overrides:

delete in class org.apache.hadoop.fs.FileSystem

Parameters:

path - path to delete

Returns:

true if one or more files/directories were deleted; false otherwise

Throws:

IOException

delete

public boolean delete(org.apache.hadoop.fs.Path path,
                      boolean recursive)
               throws IOException

Attempts to delete the file or directory with the specified path.

Specified by:

delete in class org.apache.hadoop.fs.FileSystem

Parameters:

path - path to delete

recursive - if true, will attempt to delete all children of the path

Returns:

true if one or more files/directories were deleted; false otherwise

Throws:

IOException

getDefaultBlockSize

public long getDefaultBlockSize()

Overrides:

getDefaultBlockSize in class org.apache.hadoop.fs.FileSystem

getFileBlockLocations

@Nullable
public org.apache.hadoop.fs.BlockLocation[] getFileBlockLocations(org.apache.hadoop.fs.FileStatus file,
                                                                            long start,
                                                                            long len)
                                                                     throws IOException

Overrides:

getFileBlockLocations in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

getDefaultReplication

public short getDefaultReplication()

Overrides:

getDefaultReplication in class org.apache.hadoop.fs.FileSystem

setReplication

public boolean setReplication(org.apache.hadoop.fs.Path path,
                              short replication)
                       throws IOException

Overrides:

setReplication in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

getFileStatus

public org.apache.hadoop.fs.FileStatus getFileStatus(org.apache.hadoop.fs.Path path)
                                              throws IOException

If the file does not exist in Alluxio, query it from HDFS.

Specified by:

getFileStatus in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

setOwner

public void setOwner(org.apache.hadoop.fs.Path path,
                     String username,
                     String groupname)
              throws IOException

Changes owner or group of a path (i.e. a file or a directory). If username is null, the original username remains unchanged. Same as groupname. If username and groupname are non-null, both of them will be changed.

Overrides:

setOwner in class org.apache.hadoop.fs.FileSystem

Parameters:

path - path to set owner or group

username - username to be set

groupname - groupname to be set

Throws:

IOException

setPermission

public void setPermission(org.apache.hadoop.fs.Path path,
                          org.apache.hadoop.fs.permission.FsPermission permission)
                   throws IOException

Changes permission of a path.

Overrides:

setPermission in class org.apache.hadoop.fs.FileSystem

Parameters:

path - path to set permission

permission - permission set to path

Throws:

IOException

getUri

public URI getUri()

Specified by:

getUri in class org.apache.hadoop.fs.FileSystem

getWorkingDirectory

public org.apache.hadoop.fs.Path getWorkingDirectory()

Specified by:

getWorkingDirectory in class org.apache.hadoop.fs.FileSystem

initialize

public void initialize(URI uri,
                       org.apache.hadoop.conf.Configuration conf)
                throws IOException

Sets up a lazy connection to Alluxio through mFileSystem. This must be called before client operations. If it is called twice on the same object concurrently, one of the thread will do the initialization work, the other will wait until initialization is done. If it is called after initialized, then this is a noop.

Overrides:

initialize in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

initialize

public void initialize(URI uri,
                       org.apache.hadoop.conf.Configuration conf,
                       @Nullable
                       AlluxioConfiguration alluxioConfiguration)
                throws IOException

Initialize the FileSystem.

Parameters:

uri - file system Uri

conf - hadoop configuration

alluxioConfiguration - [optional] alluxio configuration

Throws:

IOException

listStatus

public org.apache.hadoop.fs.FileStatus[] listStatus(org.apache.hadoop.fs.Path path)
                                             throws IOException

Specified by:

listStatus in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

mkdirs

public boolean mkdirs(org.apache.hadoop.fs.Path path,
                      org.apache.hadoop.fs.permission.FsPermission permission)
               throws IOException

Attempts to create a folder with the specified path. Parent directories will be created.

Specified by:

mkdirs in class org.apache.hadoop.fs.FileSystem

Parameters:

path - path to create

permission - permissions to grant the created folder

Returns:

true if the indicated folder is created successfully or already exists

Throws:

IOException

open

public org.apache.hadoop.fs.FSDataInputStream open(org.apache.hadoop.fs.Path path,
                                                   int bufferSize)
                                            throws IOException

Attempts to open the specified file for reading.

Specified by:

open in class org.apache.hadoop.fs.FileSystem

Parameters:

path - the file name to open

bufferSize - stream buffer size in bytes, currently unused

Returns:

an FSDataInputStream at the indicated path of a file

Throws:

IOException

rename

public boolean rename(org.apache.hadoop.fs.Path src,
                      org.apache.hadoop.fs.Path dst)
               throws IOException

Specified by:

rename in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

setWorkingDirectory

public void setWorkingDirectory(org.apache.hadoop.fs.Path path)

Specified by:

setWorkingDirectory in class org.apache.hadoop.fs.FileSystem