alluxio.underfs.obs

Class OBSUnderFileSystem

java.lang.Object

alluxio.underfs.BaseUnderFileSystem

alluxio.underfs.ObjectUnderFileSystem

alluxio.underfs.obs.OBSUnderFileSystem

All Implemented Interfaces:

AsyncUfsClient, UnderFileSystem, Closeable, AutoCloseable

@ThreadSafe
public class OBSUnderFileSystem
extends ObjectUnderFileSystem

Huawei OBS UnderFileSystem implementation.

Nested Class Summary

Nested classes/interfaces inherited from class alluxio.underfs.ObjectUnderFileSystem

ObjectUnderFileSystem.DeleteBuffer, ObjectUnderFileSystem.ObjectListingChunk, ObjectUnderFileSystem.ObjectPermissions, ObjectUnderFileSystem.ObjectStatus, ObjectUnderFileSystem.ObjectStoreOperation<T>, ObjectUnderFileSystem.OperationBuffer<T>, ObjectUnderFileSystem.RenameBuffer, ObjectUnderFileSystem.UfsStatusIterator

Nested classes/interfaces inherited from interface alluxio.underfs.UnderFileSystem

UnderFileSystem.Factory, UnderFileSystem.SpaceType

Field Summary

Fields inherited from class alluxio.underfs.ObjectUnderFileSystem

mExecutorService, mRootKeySupplier, PATH_SEPARATOR, PATH_SEPARATOR_CHAR

Fields inherited from class alluxio.underfs.BaseUnderFileSystem

EMPTY_ACL, mUfsConf, mUri

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	OBSUnderFileSystem(AlluxioURI uri, com.obs.services.ObsClient obsClient, String bucketName, String bucketType, UnderFileSystemConfiguration conf) Constructor for OBSUnderFileSystem.

Method Summary

Modifier and Type	Method and Description
void	cleanup() Cleans up the under file system.
protected boolean	copyObject(String src, String dst) Copies an object to another key.
boolean	createEmptyObject(String key) Creates a zero-byte object used to encode a directory.
static OBSUnderFileSystem	createInstance(AlluxioURI uri, UnderFileSystemConfiguration conf) Constructs a new instance of OBSUnderFileSystem.
protected OutputStream	createObject(String key) Creates an OutputStream for object uploads.
protected boolean	deleteObject(String key) Internal function to delete a key.
protected List<String>	deleteObjects(List<String> keys) Internal function to delete a list of keys.
protected String	getFolderSuffix() Get suffix used to encode a directory.
protected com.obs.services.model.ObjectListing	getObjectListingChunk(com.obs.services.model.ListObjectsRequest request)
protected ObjectUnderFileSystem.ObjectListingChunk	getObjectListingChunk(String key, boolean recursive) Gets a (partial) object listing result for the given key.
protected ObjectUnderFileSystem.ObjectStatus	getObjectStatus(String key) Get metadata information about object.
protected ObjectUnderFileSystem.ObjectPermissions	getPermissions() Permissions for the mounted bucket.
protected String	getRootKey() Get full path of root in object store.
UfsStatus	getStatus(String path) Gets the file or directory status.
String	getUnderFSType() Returns the name of the under filesystem implementation.
boolean	isDirectory(String path) Checks if a directory exists in under file system.
protected InputStream	openObject(String key, OpenOptions options, RetryPolicy retryPolicy) Internal function to open an input stream to an object.
PositionReader	openPositionRead(String path, long fileLength) Opens a file for position read.
boolean	renameDirectory(String src, String dst) Renames a directory from src to dst in under file system.
void	setMode(String path, short mode) Changes posix file mode.
void	setOwner(String path, String user, String group) Sets the user and group of the given path.

Methods inherited from class alluxio.underfs.ObjectUnderFileSystem

close, connectFromMaster, connectFromWorker, convertToFolderName, create, createNonexistingFile, createNonexistingFile, deleteDirectory, deleteExistingDirectory, deleteExistingDirectory, deleteExistingFile, deleteFile, getBlockSizeByte, getChildName, getDirectoryStatus, getExistingDirectoryStatus, getExistingFileStatus, getExistingStatus, getFileLocations, getFileLocations, getFileStatus, getListingChunkLength, getListingChunkLengthMax, getObjectListingChunk, getObjectListingChunkForPath, getObjectListingChunkForPath, getParentPath, getSpace, getStatus, isExistingDirectory, isFile, isObjectStorage, isRoot, listInternal, listStatus, listStatus, listStatusIterable, mkdirs, mkdirsInternal, open, openExistingFile, openExistingFile, parentExists, renameFile, renameRenamableDirectory, renameRenamableFile, retryOnException, stripPrefixIfPresent, supportsFlush

Methods inherited from class alluxio.underfs.BaseUnderFileSystem

create, deleteDirectory, exists, getAclPair, getConfiguration, getFingerprint, getOperationMode, getParsedFingerprint, getParsedFingerprint, getPhysicalStores, isSeekable, mkdirs, open, performListingAsync, resolveUri, setAclEntries, validatePath

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface alluxio.underfs.UnderFileSystem

getFileStatus, getTemporaryToken, isTemporaryConnectionToken, listStatuses, refreshTemporaryToken, setTemporaryTokenRefreshContext

Constructor Detail

OBSUnderFileSystem

protected OBSUnderFileSystem(AlluxioURI uri,
                             com.obs.services.ObsClient obsClient,
                             String bucketName,
                             String bucketType,
                             UnderFileSystemConfiguration conf)

Constructor for OBSUnderFileSystem.

Parameters:

uri - the AlluxioURI for this UFS

obsClient - Huawei OBS client

bucketName - bucket name of user's configured Alluxio bucket

conf - configuration for this UFS

Method Detail

createInstance

public static OBSUnderFileSystem createInstance(AlluxioURI uri,
                                                UnderFileSystemConfiguration conf)

Constructs a new instance of OBSUnderFileSystem.

Parameters:

uri - the AlluxioURI for this UFS

conf - the configuration for this UFS

Returns:

the created OBSUnderFileSystem instance

cleanup

public void cleanup()

Description copied from interface: UnderFileSystem

Cleans up the under file system. If any data or files are created and not completed/aborted correctly in normal ways, they should be cleaned in this method.

Specified by:

cleanup in interface UnderFileSystem

Overrides:

cleanup in class ObjectUnderFileSystem

getUnderFSType

public String getUnderFSType()

Description copied from interface: UnderFileSystem

Returns the name of the under filesystem implementation. The name should be lowercase and not include any spaces, e.g. "hdfs", "s3".

Returns:

name of the under filesystem implementation

openPositionRead

public PositionReader openPositionRead(String path,
                                       long fileLength)

Description copied from interface: UnderFileSystem

Opens a file for position read.

Parameters:

path - the path to read

fileLength - the file length

Returns:

the position reader

setOwner

public void setOwner(String path,
                     String user,
                     String group)

Description copied from interface: UnderFileSystem

Sets the user and group of the given path. An empty implementation should be provided if unsupported.

Parameters:

path - the path of the file

user - the new owner to set, unchanged if null

group - the new group to set, unchanged if null

setMode

public void setMode(String path,
                    short mode)
             throws IOException

Description copied from interface: UnderFileSystem

Changes posix file mode.

Parameters:

path - the path of the file

mode - the mode to set in short format, e.g. 0777

Throws:

IOException

copyObject

protected boolean copyObject(String src,
                             String dst)

Description copied from class: ObjectUnderFileSystem

Copies an object to another key.

Specified by:

copyObject in class ObjectUnderFileSystem

Parameters:

src - the source key to copy

dst - the destination key to copy to

Returns:

true if the operation was successful, false otherwise

createEmptyObject

public boolean createEmptyObject(String key)

Description copied from class: ObjectUnderFileSystem

Creates a zero-byte object used to encode a directory.

Specified by:

createEmptyObject in class ObjectUnderFileSystem

Parameters:

key - the key to create

Returns:

true if the operation was successful

createObject

protected OutputStream createObject(String key)
                             throws IOException

Description copied from class: ObjectUnderFileSystem

Creates an OutputStream for object uploads.

Specified by:

createObject in class ObjectUnderFileSystem

Parameters:

key - ufs key including scheme and bucket

Returns:

new OutputStream

Throws:

IOException

deleteObject

protected boolean deleteObject(String key)

Description copied from class: ObjectUnderFileSystem

Internal function to delete a key.

Specified by:

deleteObject in class ObjectUnderFileSystem

Parameters:

key - the key to delete

Returns:

true if successful, false if an exception is thrown

deleteObjects

protected List<String> deleteObjects(List<String> keys)
                              throws IOException

Description copied from class: ObjectUnderFileSystem

Internal function to delete a list of keys.

Overrides:

deleteObjects in class ObjectUnderFileSystem

Parameters:

keys - the list of keys to delete

Returns:

list of successfully deleted keys

Throws:

IOException

getFolderSuffix

protected String getFolderSuffix()

Description copied from class: ObjectUnderFileSystem

Get suffix used to encode a directory.

Specified by:

getFolderSuffix in class ObjectUnderFileSystem

Returns:

folder suffix

getObjectListingChunk

protected ObjectUnderFileSystem.ObjectListingChunk getObjectListingChunk(String key,
                                                                         boolean recursive)
                                                                  throws IOException

Description copied from class: ObjectUnderFileSystem

Gets a (partial) object listing result for the given key.

Specified by:

getObjectListingChunk in class ObjectUnderFileSystem

Parameters:

key - pseudo-directory key excluding header and bucket

recursive - whether to request immediate children only, or all descendants

Returns:

chunked object listing, or null if key is not found

Throws:

IOException

getObjectListingChunk

protected com.obs.services.model.ObjectListing getObjectListingChunk(com.obs.services.model.ListObjectsRequest request)

getObjectStatus

protected ObjectUnderFileSystem.ObjectStatus getObjectStatus(String key)

Description copied from class: ObjectUnderFileSystem

Get metadata information about object. Implementations should process the key as is, which may be a file or a directory key.

Specified by:

getObjectStatus in class ObjectUnderFileSystem

Parameters:

key - ufs key to get metadata for

Returns:

ObjectUnderFileSystem.ObjectStatus if key exists and successful, otherwise null

getStatus

public UfsStatus getStatus(String path)
                    throws IOException

Description copied from interface: UnderFileSystem

Gets the file or directory status. The caller does not need to know if the path is a file or directory. This method will determine the path type, and will return the appropriate status.

Parameters:

path - the path to get the status

Returns:

the file or directory status

Throws:

FileNotFoundException - when the path does not exist

IOException

isDirectory

public boolean isDirectory(String path)
                    throws IOException

Description copied from interface: UnderFileSystem

Checks if a directory exists in under file system.

Specified by:

isDirectory in interface UnderFileSystem

Overrides:

isDirectory in class ObjectUnderFileSystem

Parameters:

path - the absolute directory path

Returns:

true if the path exists and is a directory, false otherwise

Throws:

IOException

getPermissions

protected ObjectUnderFileSystem.ObjectPermissions getPermissions()

Description copied from class: ObjectUnderFileSystem

Permissions for the mounted bucket.

Specified by:

getPermissions in class ObjectUnderFileSystem

Returns:

permissions

getRootKey

protected String getRootKey()

Description copied from class: ObjectUnderFileSystem

Get full path of root in object store.

Specified by:

getRootKey in class ObjectUnderFileSystem

Returns:

full path including scheme and bucket

openObject

protected InputStream openObject(String key,
                                 OpenOptions options,
                                 RetryPolicy retryPolicy)
                          throws IOException

Description copied from class: ObjectUnderFileSystem

Internal function to open an input stream to an object.

Specified by:

openObject in class ObjectUnderFileSystem

Parameters:

key - the key to open

options - the open options

retryPolicy - the retry policy of the opened stream to solve eventual consistency issue

Returns:

an InputStream to read from key

Throws:

IOException

renameDirectory

public boolean renameDirectory(String src,
                               String dst)
                        throws IOException

Description copied from interface: UnderFileSystem

Renames a directory from src to dst in under file system.

Specified by:

renameDirectory in interface UnderFileSystem

Overrides:

renameDirectory in class ObjectUnderFileSystem

Parameters:

src - the source directory path

dst - the destination directory path

Returns:

true if succeed, false otherwise

Throws:

IOException