ObjectUnderFileSystem (Alluxio Parent 305-SNAPSHOT API)

java.lang.Object
- alluxio.underfs.BaseUnderFileSystem
- - alluxio.underfs.ObjectUnderFileSystem

All Implemented Interfaces:

AsyncUfsClient, UnderFileSystem, Closeable, AutoCloseable

Direct Known Subclasses:

COSUnderFileSystem, GCSUnderFileSystem, GCSV2UnderFileSystem, OBSUnderFileSystem, OSSUnderFileSystem, S3AUnderFileSystem
```
@ThreadSafe
public abstract class ObjectUnderFileSystem
extends BaseUnderFileSystem
```
An object based abstract UnderFileSystem. Object Stores implementing the UnderFileSystem interface should derive from this class.

Nested Class Summary

Nested Classes
Modifier and Type	Class and Description
`protected class`	`ObjectUnderFileSystem.DeleteBuffer` Object keys added to a `ObjectUnderFileSystem.DeleteBuffer` will be deleted in batches.
`static interface`	`ObjectUnderFileSystem.ObjectListingChunk` A chunk of listing results.
`static class`	`ObjectUnderFileSystem.ObjectPermissions` Permissions in object UFS.
`protected static class`	`ObjectUnderFileSystem.ObjectStatus` Information about a single object in object UFS.
`protected static interface`	`ObjectUnderFileSystem.ObjectStoreOperation<T>` Represents an object store operation.
`protected class`	`ObjectUnderFileSystem.OperationBuffer<T>` Operations added to this buffer are performed concurrently.
`protected class`	`ObjectUnderFileSystem.RenameBuffer` File paths added to a `ObjectUnderFileSystem.RenameBuffer` will be renamed concurrently.
`class`	`ObjectUnderFileSystem.UfsStatusIterator` The UFS status iterator that iterates the ufs statuses and fetches the chunk by lazy.

Nested classes/interfaces inherited from interface alluxio.underfs.UnderFileSystem
UnderFileSystem.Factory, UnderFileSystem.SpaceType

Field Summary

Fields
Modifier and Type	Field and Description
`protected ExecutorService`	`mExecutorService` Executor service used for parallel UFS operations such as bulk deletes.
`protected java.util.function.Supplier<String>`	`mRootKeySupplier` The root key of an object fs.
`protected static String`	`PATH_SEPARATOR` Value used to indicate nested structure.
`protected static char`	`PATH_SEPARATOR_CHAR` Value used to indicate nested structure.

Fields inherited from class alluxio.underfs.BaseUnderFileSystem
EMPTY_ACL, mUfsConf, mUri

Constructor Summary

Constructors
Modifier	Constructor and Description
`protected`	`ObjectUnderFileSystem(AlluxioURI uri, UnderFileSystemConfiguration ufsConf)` Constructs an `ObjectUnderFileSystem`.

Method Summary

All Methods Instance Methods Abstract Methods Concrete Methods
Modifier and Type	Method and Description
`void`	`cleanup()` Cleans up the under file system.
`void`	`close()`
`void`	`connectFromMaster(String hostname)` Takes any necessary actions required to establish a connection to the under file system from the given master host e.g.
`void`	`connectFromWorker(String hostname)` Takes any necessary actions required to establish a connection to the under file system from the given worker host e.g.
`protected String`	`convertToFolderName(String key)` Appends the directory suffix to the key.
`protected abstract boolean`	`copyObject(String src, String dst)` Copies an object to another key.
`OutputStream`	`create(String path, CreateOptions options)` Creates a file in the under file system with the specified `CreateOptions`.
`abstract boolean`	`createEmptyObject(String key)` Creates a zero-byte object used to encode a directory.
`OutputStream`	`createNonexistingFile(String path)` Creates a file in the under file system with the indicated name.
`OutputStream`	`createNonexistingFile(String path, CreateOptions options)` Creates a file in the under file system with the specified `CreateOptions`.
`protected abstract OutputStream`	`createObject(String key)` Creates an `OutputStream` for object uploads.
`boolean`	`deleteDirectory(String path, DeleteOptions options)` Deletes a directory from the under file system with the indicated name.
`boolean`	`deleteExistingDirectory(String path)` Deletes a directory from the under file system.
`boolean`	`deleteExistingDirectory(String path, DeleteOptions options)` Deletes a directory from the under file system with the indicated name.
`boolean`	`deleteExistingFile(String path)` Deletes a file from the under file system with the indicated name.
`boolean`	`deleteFile(String path)` Deletes a file from the under file system with the indicated name.
`protected abstract boolean`	`deleteObject(String key)` Internal function to delete a key.
`protected List<String>`	`deleteObjects(List<String> keys)` Internal function to delete a list of keys.
`long`	`getBlockSizeByte(String path)` Gets the block size in bytes.
`protected String`	`getChildName(String child, String parent)` Gets the child name based on the parent name.
`UfsDirectoryStatus`	`getDirectoryStatus(String path)` Gets the directory status.
`UfsDirectoryStatus`	`getExistingDirectoryStatus(String path)` Gets the directory status.
`UfsFileStatus`	`getExistingFileStatus(String path)` Gets the file status.
`UfsStatus`	`getExistingStatus(String path)` Gets the file or directory status.
`List<String>`	`getFileLocations(String path)` Gets the list of locations of the indicated path.
`List<String>`	`getFileLocations(String path, FileLocationOptions options)` Gets the list of locations of the indicated path given options.
`UfsFileStatus`	`getFileStatus(String path, GetStatusOptions options)` Gets the file status.
`protected abstract String`	`getFolderSuffix()` Get suffix used to encode a directory.
`protected int`	`getListingChunkLength(AlluxioConfiguration conf)` The number of items to query in a single listing chunk.
`protected int`	`getListingChunkLengthMax()` Maximum number of items in a single listing chunk supported by the under store.
`protected abstract ObjectUnderFileSystem.ObjectListingChunk`	`getObjectListingChunk(String key, boolean recursive)` Gets a (partial) object listing result for the given key.
`protected ObjectUnderFileSystem.ObjectListingChunk`	`getObjectListingChunk(String key, boolean recursive, String startAfter, int batchSize)` Gets a (partial) object listing result for the given key.
`protected ObjectUnderFileSystem.ObjectListingChunk`	`getObjectListingChunkForPath(String path, boolean recursive)`
`protected ObjectUnderFileSystem.ObjectListingChunk`	`getObjectListingChunkForPath(String path, boolean recursive, String startAfter, int batchSize)` Gets a (partial) object listing for the given path.
`protected abstract ObjectUnderFileSystem.ObjectStatus`	`getObjectStatus(String key)` Get metadata information about object.
`protected String`	`getParentPath(String path)` Get parent path.
`protected abstract ObjectUnderFileSystem.ObjectPermissions`	`getPermissions()` Permissions for the mounted bucket.
`protected abstract String`	`getRootKey()` Get full path of root in object store.
`long`	`getSpace(String path, UnderFileSystem.SpaceType type)` Queries the under file system about the space of the indicated path (e.g., space left, space used and etc).
`UfsStatus`	`getStatus(String path, GetStatusOptions options)` Gets the file or directory status.
`boolean`	`isDirectory(String path)` Checks if a directory exists in under file system.
`boolean`	`isExistingDirectory(String path)` Checks if a directory exists in under file system.
`boolean`	`isFile(String path)` Checks if a file exists in under file system.
`boolean`	`isObjectStorage()`
`protected boolean`	`isRoot(String path)` Checks if the path is the root.
`protected UfsStatus[]`	`listInternal(String path, ListOptions options)` Lists the files in the given path, the paths will be their logical names and not contain the folder suffix.
`UfsStatus[]`	`listStatus(String path)` Returns an array of statuses of the files and directories in the directory denoted by this abstract pathname.
`UfsStatus[]`	`listStatus(String path, ListOptions options)` Returns an array of statuses of the files and directories in the directory denoted by this abstract pathname, with options.
`Iterator<UfsStatus>`	`listStatusIterable(String path, ListOptions options, String startAfter, int batchSize)` Lists the ufs statuses iteratively.
`boolean`	`mkdirs(String path, MkdirsOptions options)` Creates the directory named by this abstract pathname, with specified `MkdirsOptions`.
`protected boolean`	`mkdirsInternal(String key)` Creates a directory flagged file with the key and folder suffix.
`InputStream`	`open(String path, OpenOptions options)` Opens an `InputStream` for a file in under filesystem at the indicated path.
`InputStream`	`openExistingFile(String path)` Opens an `InputStream` for a file in under filesystem at the indicated path.
`InputStream`	`openExistingFile(String path, OpenOptions options)` Opens an `InputStream` for a file in under filesystem at the indicated path.
`protected abstract InputStream`	`openObject(String key, OpenOptions options, RetryPolicy retryPolicy)` Internal function to open an input stream to an object.
`protected boolean`	`parentExists(String path)` Treating the object store as a file system, checks if the parent directory exists.
`boolean`	`renameDirectory(String src, String dst)` Renames a directory from `src` to `dst` in under file system.
`boolean`	`renameFile(String src, String dst)` Renames a file from `src` to `dst` in under file system.
`boolean`	`renameRenamableDirectory(String src, String dst)` Renames a directory from `src` to `dst` in under file system.
`boolean`	`renameRenamableFile(String src, String dst)` Renames a file from `src` to `dst` in under file system.
`protected <T> T`	`retryOnException(ObjectUnderFileSystem.ObjectStoreOperation<T> op, java.util.function.Supplier<String> description)` Retries the given object store operation when it throws exceptions to resolve eventual consistency issue.
`String`	`stripPrefixIfPresent(String path)` Strips the bucket prefix or the preceding path separator from the path if it is present.
`boolean`	`supportsFlush()` Whether this type of UFS supports flush.

Methods inherited from class alluxio.underfs.BaseUnderFileSystem
create, deleteDirectory, exists, getAclPair, getConfiguration, getFingerprint, getOperationMode, getParsedFingerprint, getParsedFingerprint, getPhysicalStores, isSeekable, mkdirs, open, performListingAsync, resolveUri, setAclEntries, validatePath

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface alluxio.underfs.UnderFileSystem
getFileStatus, getStatus, getTemporaryToken, getUnderFSType, isTemporaryConnectionToken, listStatuses, openPositionRead, refreshTemporaryToken, setMode, setOwner, setTemporaryTokenRefreshContext

- Field Detail
  - PATH_SEPARATOR_CHAR
```
protected static final char PATH_SEPARATOR_CHAR
```
    Value used to indicate nested structure.
    
    See Also:
    
    Constant Field Values
  - PATH_SEPARATOR
```
protected static final String PATH_SEPARATOR
```
    Value used to indicate nested structure. This is a string representation of PATH_SEPARATOR_CHAR.
  - mExecutorService
```
protected ExecutorService mExecutorService
```
    Executor service used for parallel UFS operations such as bulk deletes.
  - mRootKeySupplier
```
protected final java.util.function.Supplier<String> mRootKeySupplier
```
    The root key of an object fs.
- Constructor Detail
  - ObjectUnderFileSystem
```
protected ObjectUnderFileSystem(AlluxioURI uri,
                                UnderFileSystemConfiguration ufsConf)
```
    Constructs an ObjectUnderFileSystem.
    
    Parameters:
    
    uri - the AlluxioURI used to create this ufs
    
    ufsConf - UFS configuration
- Method Detail
  - cleanup
```
public void cleanup()
             throws IOException
```
    Description copied from interface: UnderFileSystem
    
    Cleans up the under file system. If any data or files are created and not completed/aborted correctly in normal ways, they should be cleaned in this method.
    
    Throws:
    
    IOException
  - close
```
public void close()
           throws IOException
```
    Specified by:
    
    close in interface Closeable
    
    Specified by:
    
    close in interface AutoCloseable
    
    Overrides:
    
    close in class BaseUnderFileSystem
    
    Throws:
    
    IOException
  - connectFromMaster
```
public void connectFromMaster(String hostname)
```
    Description copied from interface: UnderFileSystem
    
    Takes any necessary actions required to establish a connection to the under file system from the given master host e.g. logging in
    Depending on the implementation this may be a no-op
    
    Parameters:
    
    hostname - the host that wants to connect to the under file system
  - connectFromWorker
```
public void connectFromWorker(String hostname)
```
    Description copied from interface: UnderFileSystem
    
    Takes any necessary actions required to establish a connection to the under file system from the given worker host e.g. logging in
    Depending on the implementation this may be a no-op
    
    Parameters:
    
    hostname - the host that wants to connect to the under file system
  - create
```
public OutputStream create(String path,
                           CreateOptions options)
                    throws IOException
```
    Description copied from interface: UnderFileSystem
    
    Creates a file in the under file system with the specified CreateOptions. Implementations should make sure that the path under creation appears in listings only after a successful close and that contents are written in its entirety or not at all.
    
    Parameters:
    
    path - the file name
    
    options - the options for create
    
    Returns:
    
    A OutputStream object
    
    Throws:
    
    IOException
  - createNonexistingFile
```
public OutputStream createNonexistingFile(String path)
                                   throws IOException
```
    Description copied from interface: UnderFileSystem
    
    Creates a file in the under file system with the indicated name. Similar to UnderFileSystem.create(String) but deals with the delete-then-create eventual consistency issue.
    
    Parameters:
    
    path - the file name
    
    Returns:
    
    A OutputStream object
    
    Throws:
    
    IOException
  - createNonexistingFile
```
public OutputStream createNonexistingFile(String path,
                                          CreateOptions options)
                                   throws IOException
```
    Description copied from interface: UnderFileSystem
    
    Creates a file in the under file system with the specified CreateOptions. Similar to UnderFileSystem.create(String, CreateOptions) but deals with the delete-then-create eventual consistency issue.
    
    Parameters:
    
    path - the file name
    
    options - the options for create
    
    Returns:
    
    A OutputStream object
    
    Throws:
    
    IOException
  - deleteFile
```
public boolean deleteFile(String path)
                   throws IOException
```
    Description copied from interface: UnderFileSystem
    
    Deletes a file from the under file system with the indicated name.
    
    Parameters:
    
    path - of the file to delete
    
    Returns:
    
    true if file was found and deleted, false otherwise
    
    Throws:
    
    IOException
  - deleteExistingFile
```
public boolean deleteExistingFile(String path)
                           throws IOException
```
    Description copied from interface: UnderFileSystem
    
    Deletes a file from the under file system with the indicated name. Similar to UnderFileSystem.deleteFile(String) but deals with the create-then-delete eventual consistency issue.
    
    Parameters:
    
    path - of the file to delete
    
    Returns:
    
    true if file was found and deleted, false otherwise
    
    Throws:
    
    IOException
  - deleteDirectory
```
public boolean deleteDirectory(String path,
                               DeleteOptions options)
                        throws IOException
```
    Description copied from interface: UnderFileSystem
    
    Deletes a directory from the under file system with the indicated name.
    
    Parameters:
    
    path - of the directory to delete
    
    options - for directory delete semantics
    
    Returns:
    
    true if directory was found and deleted, false otherwise
    
    Throws:
    
    IOException
  - deleteExistingDirectory
```
public boolean deleteExistingDirectory(String path)
                                throws IOException
```
    Description copied from interface: UnderFileSystem
    
    Deletes a directory from the under file system. Similar to UnderFileSystem.deleteDirectory(String) but deals with the create-delete eventual consistency issue.
    
    Parameters:
    
    path - of the directory to delete
    
    Returns:
    
    true if directory was found and deleted, false otherwise
    
    Throws:
    
    IOException
  - deleteExistingDirectory
```
public boolean deleteExistingDirectory(String path,
                                       DeleteOptions options)
                                throws IOException
```
    Description copied from interface: UnderFileSystem
    
    Deletes a directory from the under file system with the indicated name. Similar to UnderFileSystem.deleteDirectory(String, DeleteOptions) but deals with the create-then-delete eventual consistency issue.
    
    Parameters:
    
    path - of the directory to delete
    
    options - for directory delete semantics
    
    Returns:
    
    true if directory was found and deleted, false otherwise
    
    Throws:
    
    IOException
  - getBlockSizeByte
```
public long getBlockSizeByte(String path)
                      throws IOException
```
    Gets the block size in bytes. This method defaults to the default user block size in Alluxio.
    
    Parameters:
    
    path - the file name
    
    Returns:
    
    the default Alluxio user block size
    
    Throws:
    
    IOException
  - getDirectoryStatus
```
public UfsDirectoryStatus getDirectoryStatus(String path)
                                      throws IOException
```
    Description copied from interface: UnderFileSystem
    
    Gets the directory status. The caller must already know the path is a directory. This method will throw an exception if the path exists, but is a file.
    
    Parameters:
    
    path - the path to the directory
    
    Returns:
    
    the directory status
    
    Throws:
    
    FileNotFoundException - when the path does not exist
    
    IOException
  - getExistingDirectoryStatus
```
public UfsDirectoryStatus getExistingDirectoryStatus(String path)
                                              throws IOException
```
    Description copied from interface: UnderFileSystem
    
    Gets the directory status. Similar to UnderFileSystem.getDirectoryStatus(String) but deals with the write-then-get-status eventual consistency issue.
    
    Parameters:
    
    path - the path to the directory
    
    Returns:
    
    the directory status
    
    Throws:
    
    IOException
  - getFileLocations
```
@Nullable
public List<String> getFileLocations(String path)
                                        throws IOException
```
    Description copied from interface: UnderFileSystem
    
    Gets the list of locations of the indicated path.
    
    Parameters:
    
    path - the file name
    
    Returns:
    
    The list of locations
    
    Throws:
    
    IOException
  - getFileLocations
```
@Nullable
public List<String> getFileLocations(String path,
                                               FileLocationOptions options)
                                        throws IOException
```
    Description copied from interface: UnderFileSystem
    
    Gets the list of locations of the indicated path given options.
    
    Parameters:
    
    path - the file name
    
    options - method options
    
    Returns:
    
    The list of locations
    
    Throws:
    
    IOException
  - getSpace
```
public long getSpace(String path,
                     UnderFileSystem.SpaceType type)
              throws IOException
```
    Description copied from interface: UnderFileSystem
    
    Queries the under file system about the space of the indicated path (e.g., space left, space used and etc).
    
    Parameters:
    
    path - the path to query
    
    type - the type of queries
    
    Returns:
    
    The space in bytes
    
    Throws:
    
    IOException
  - getFileStatus
```
public UfsFileStatus getFileStatus(String path,
                                   GetStatusOptions options)
                            throws IOException
```
    Description copied from interface: UnderFileSystem
    
    Gets the file status. The caller must already know the path is a file. This method will throw an exception if the path exists, but is a directory.
    
    Parameters:
    
    path - the path to the file
    
    options - method options
    
    Returns:
    
    the file status
    
    Throws:
    
    FileNotFoundException - when the path does not exist
    
    IOException
  - getExistingFileStatus
```
public UfsFileStatus getExistingFileStatus(String path)
                                    throws IOException
```
    Description copied from interface: UnderFileSystem
    
    Gets the file status. Similar to UnderFileSystem.getFileStatus(String) but deals with the write-then-get-status eventual consistency issue.
    
    Parameters:
    
    path - the path to the file
    
    Returns:
    
    the file status
    
    Throws:
    
    IOException
  - getStatus
```
public UfsStatus getStatus(String path,
                           GetStatusOptions options)
                    throws IOException
```
    Description copied from interface: UnderFileSystem
    
    Gets the file or directory status. The caller does not need to know if the path is a file or directory. This method will determine the path type, and will return the appropriate status.
    
    Parameters:
    
    path - the path to get the status
    
    options - method options
    
    Returns:
    
    the file or directory status
    
    Throws:
    
    FileNotFoundException - when the path does not exist
    
    IOException
  - getExistingStatus
```
public UfsStatus getExistingStatus(String path)
                            throws IOException
```
    Description copied from interface: UnderFileSystem
    
    Gets the file or directory status. Similar to UnderFileSystem.getStatus(String) but deals with the write-then-get-status eventual consistency issue.
    
    Parameters:
    
    path - the path to get the status
    
    Returns:
    
    the file or directory status
    
    Throws:
    
    IOException
  - isDirectory
```
public boolean isDirectory(String path)
                    throws IOException
```
    Description copied from interface: UnderFileSystem
    
    Checks if a directory exists in under file system.
    
    Parameters:
    
    path - the absolute directory path
    
    Returns:
    
    true if the path exists and is a directory, false otherwise
    
    Throws:
    
    IOException
  - isExistingDirectory
```
public boolean isExistingDirectory(String path)
                            throws IOException
```
    Description copied from interface: UnderFileSystem
    
    Checks if a directory exists in under file system. Similar to UnderFileSystem.isDirectory(String) but deals with the write-then-list eventual consistency issue.
    
    Parameters:
    
    path - the absolute directory path
    
    Returns:
    
    true if the path exists and is a directory, false otherwise
    
    Throws:
    
    IOException
  - isFile
```
public boolean isFile(String path)
               throws IOException
```
    Description copied from interface: UnderFileSystem
    
    Checks if a file exists in under file system.
    
    Parameters:
    
    path - the absolute file path
    
    Returns:
    
    true if the path exists and is a file, false otherwise
    
    Throws:
    
    IOException
  - isObjectStorage
```
public boolean isObjectStorage()
```
    Specified by:
    
    isObjectStorage in interface UnderFileSystem
    
    Overrides:
    
    isObjectStorage in class BaseUnderFileSystem
    
    Returns:
    
    true if under storage is an object store, false otherwise
  - listStatus
```
public UfsStatus[] listStatus(String path)
                       throws IOException
```
    Description copied from interface: UnderFileSystem
    
    Returns an array of statuses of the files and directories in the directory denoted by this abstract pathname.
    If this abstract pathname does not denote a directory, then this method returns null. Otherwise an array of statuses is returned, one for each file or directory in the directory. Names denoting the directory itself and the directory's parent directory are not included in the result. Each string is a file name rather than a complete path.
    There is no guarantee that the name strings in the resulting array will appear in any specific order; they are not, in particular, guaranteed to appear in alphabetical order.
    
    Parameters:
    
    path - the abstract pathname to list
    
    Returns:
    
    An array with the statuses of the files and directories in the directory denoted by this abstract pathname. The array will be empty if the directory is empty. Returns null if this abstract pathname does not denote a directory.
    
    Throws:
    
    IOException
  - listStatus
```
public UfsStatus[] listStatus(String path,
                              ListOptions options)
                       throws IOException
```
    Description copied from interface: UnderFileSystem
    
    Returns an array of statuses of the files and directories in the directory denoted by this abstract pathname, with options.
    If this abstract pathname does not denote a directory, then this method returns null. Otherwise an array of statuses is returned, one for each file or directory. Names denoting the directory itself and the directory's parent directory are not included in the result. Each string is a path relative to the given directory.
    There is no guarantee that the name strings in the resulting array will appear in any specific order; they are not, in particular, guaranteed to appear in alphabetical order.
    
    Specified by:
    
    listStatus in interface UnderFileSystem
    
    Overrides:
    
    listStatus in class BaseUnderFileSystem
    
    Parameters:
    
    path - the abstract pathname to list
    
    options - for list directory
    
    Returns:
    
    An array of statuses naming the files and directories in the directory denoted by this abstract pathname. The array will be empty if the directory is empty. Returns null if this abstract pathname does not denote a directory.
    
    Throws:
    
    IOException
  - listStatusIterable
```
@Nullable
public Iterator<UfsStatus> listStatusIterable(String path,
                                                        ListOptions options,
                                                        String startAfter,
                                                        int batchSize)
                                                 throws IOException
```
    Description copied from interface: UnderFileSystem
    
    Lists the ufs statuses iteratively.
    
    Specified by:
    
    listStatusIterable in interface UnderFileSystem
    
    Overrides:
    
    listStatusIterable in class BaseUnderFileSystem
    
    Parameters:
    
    path - the abstract pathname to list
    
    options - for list directory
    
    startAfter - the start after token
    
    batchSize - the batch size
    
    Returns:
    
    An iterator of ufs status. Returns null if this abstract pathname does not denote a directory.
    
    Throws:
    
    IOException
  - mkdirs
```
public boolean mkdirs(String path,
                      MkdirsOptions options)
               throws IOException
```
    Description copied from interface: UnderFileSystem
    
    Creates the directory named by this abstract pathname, with specified MkdirsOptions. If the folder already exists, the method returns false.
    
    Parameters:
    
    path - the folder to create
    
    options - the options for mkdirs
    
    Returns:
    
    true if and only if the directory was created; false otherwise
    
    Throws:
    
    IOException
  - open
```
public InputStream open(String path,
                        OpenOptions options)
                 throws IOException
```
    Description copied from interface: UnderFileSystem
    
    Opens an InputStream for a file in under filesystem at the indicated path.
    
    Parameters:
    
    path - the file name
    
    options - to open input stream
    
    Returns:
    
    The InputStream object
    
    Throws:
    
    IOException
  - openExistingFile
```
public InputStream openExistingFile(String path)
                             throws IOException
```
    Description copied from interface: UnderFileSystem
    
    Opens an InputStream for a file in under filesystem at the indicated path. Similar to UnderFileSystem.open(String) but deals with the write-then-read eventual consistency issue.
    
    Parameters:
    
    path - the file name
    
    Returns:
    
    The InputStream object
    
    Throws:
    
    IOException
  - openExistingFile
```
public InputStream openExistingFile(String path,
                                    OpenOptions options)
                             throws IOException
```
    Description copied from interface: UnderFileSystem
    
    Opens an InputStream for a file in under filesystem at the indicated path. Similar to UnderFileSystem.open(String, OpenOptions) but deals with the write-then-read eventual consistency issue.
    
    Parameters:
    
    path - the file name
    
    options - to open input stream
    
    Returns:
    
    The InputStream object
    
    Throws:
    
    IOException
  - renameDirectory
```
public boolean renameDirectory(String src,
                               String dst)
                        throws IOException
```
    Description copied from interface: UnderFileSystem
    
    Renames a directory from src to dst in under file system.
    
    Parameters:
    
    src - the source directory path
    
    dst - the destination directory path
    
    Returns:
    
    true if succeed, false otherwise
    
    Throws:
    
    IOException
  - renameRenamableDirectory
```
public boolean renameRenamableDirectory(String src,
                                        String dst)
                                 throws IOException
```
    Description copied from interface: UnderFileSystem
    
    Renames a directory from src to dst in under file system. Similar to UnderFileSystem.renameDirectory(String, String) but deals with the write-src-then-rename and delete-dst-then-rename eventual consistency issue.
    
    Parameters:
    
    src - the source directory path
    
    dst - the destination directory path
    
    Returns:
    
    true if succeed, false otherwise
    
    Throws:
    
    IOException
  - renameFile
```
public boolean renameFile(String src,
                          String dst)
                   throws IOException
```
    Description copied from interface: UnderFileSystem
    
    Renames a file from src to dst in under file system.
    
    Parameters:
    
    src - the source file path
    
    dst - the destination file path
    
    Returns:
    
    true if succeed, false otherwise
    
    Throws:
    
    IOException
  - renameRenamableFile
```
public boolean renameRenamableFile(String src,
                                   String dst)
                            throws IOException
```
    Description copied from interface: UnderFileSystem
    
    Renames a file from src to dst in under file system. Similar to UnderFileSystem.renameFile(String, String) but deals with the write-src-then-rename and delete-dst-then-rename eventual consistency issue.
    
    Parameters:
    
    src - the source file path
    
    dst - the destination file path
    
    Returns:
    
    true if succeed, false otherwise
    
    Throws:
    
    IOException
  - supportsFlush
```
public boolean supportsFlush()
                      throws IOException
```
    Description copied from interface: UnderFileSystem
    
    Whether this type of UFS supports flush.
    
    Returns:
    
    true if this type of UFS supports flush, false otherwise
    
    Throws:
    
    IOException
  - createEmptyObject
```
public abstract boolean createEmptyObject(String key)
```
    Creates a zero-byte object used to encode a directory.
    
    Parameters:
    
    key - the key to create
    
    Returns:
    
    true if the operation was successful
  - createObject
```
protected abstract OutputStream createObject(String key)
                                      throws IOException
```
    Creates an OutputStream for object uploads.
    
    Parameters:
    
    key - ufs key including scheme and bucket
    
    Returns:
    
    new OutputStream
    
    Throws:
    
    IOException
  - convertToFolderName
```
protected String convertToFolderName(String key)
```
    Appends the directory suffix to the key.
    
    Parameters:
    
    key - the key to convert
    
    Returns:
    
    key as a directory path
  - copyObject
```
protected abstract boolean copyObject(String src,
                                      String dst)
                               throws IOException
```
    Copies an object to another key.
    
    Parameters:
    
    src - the source key to copy
    
    dst - the destination key to copy to
    
    Returns:
    
    true if the operation was successful, false otherwise
    
    Throws:
    
    IOException
  - deleteObject
```
protected abstract boolean deleteObject(String key)
                                 throws IOException
```
    Internal function to delete a key.
    
    Parameters:
    
    key - the key to delete
    
    Returns:
    
    true if successful, false if an exception is thrown
    
    Throws:
    
    IOException
  - deleteObjects
```
protected List<String> deleteObjects(List<String> keys)
                              throws IOException
```
    Internal function to delete a list of keys.
    
    Parameters:
    
    keys - the list of keys to delete
    
    Returns:
    
    list of successfully deleted keys
    
    Throws:
    
    IOException
  - getPermissions
```
protected abstract ObjectUnderFileSystem.ObjectPermissions getPermissions()
```
    Permissions for the mounted bucket.
    
    Returns:
    
    permissions
  - getListingChunkLengthMax
```
protected int getListingChunkLengthMax()
```
    Maximum number of items in a single listing chunk supported by the under store.
    
    Returns:
    
    the maximum length for a single listing query
  - getListingChunkLength
```
protected int getListingChunkLength(AlluxioConfiguration conf)
```
    The number of items to query in a single listing chunk.
    
    Returns:
    
    length of each list request
  - getObjectStatus
```
@Nullable
protected abstract ObjectUnderFileSystem.ObjectStatus getObjectStatus(String key)
                                                                         throws IOException
```
    Get metadata information about object. Implementations should process the key as is, which may be a file or a directory key.
    
    Parameters:
    
    key - ufs key to get metadata for
    
    Returns:
    
    ObjectUnderFileSystem.ObjectStatus if key exists and successful, otherwise null
    
    Throws:
    
    IOException
  - getParentPath
```
@Nullable
protected String getParentPath(String path)
```
    Get parent path.
    
    Parameters:
    
    path - ufs path including scheme and bucket
    
    Returns:
    
    the parent path, or null if the parent does not exist
  - isRoot
```
protected boolean isRoot(String path)
```
    Checks if the path is the root. This method supports full path (e.g. s3://bucket_name/dir) and stripped path (e.g. /dir).
    
    Parameters:
    
    path - ufs path including scheme and bucket
    
    Returns:
    
    true if the path is the root, false otherwise
  - getChildName
```
protected String getChildName(String child,
                              String parent)
                       throws IOException
```
    Gets the child name based on the parent name.
    
    Parameters:
    
    child - the key of the child
    
    parent - the key of the parent
    
    Returns:
    
    the child key with the parent prefix removed
    
    Throws:
    
    IOException
  - getFolderSuffix
```
protected abstract String getFolderSuffix()
```
    Get suffix used to encode a directory.
    
    Returns:
    
    folder suffix
  - getObjectListingChunk
```
@Nullable
protected ObjectUnderFileSystem.ObjectListingChunk getObjectListingChunk(String key,
                                                                                   boolean recursive,
                                                                                   String startAfter,
                                                                                   int batchSize)
                                                                            throws IOException
```
    Gets a (partial) object listing result for the given key.
    
    Parameters:
    
    key - pseudo-directory key excluding header and bucket
    
    recursive - whether to request immediate children only, or all descendants
    
    startAfter - indicates where the listing starts
    
    batchSize - the batch size of each chunk
    
    Returns:
    
    chunked object listing, or null if key is not found
    
    Throws:
    
    IOException
  - getObjectListingChunk
```
@Nullable
protected abstract ObjectUnderFileSystem.ObjectListingChunk getObjectListingChunk(String key,
                                                                                            boolean recursive)
                                                                                     throws IOException
```
    Gets a (partial) object listing result for the given key.
    
    Parameters:
    
    key - pseudo-directory key excluding header and bucket
    
    recursive - whether to request immediate children only, or all descendants
    
    Returns:
    
    chunked object listing, or null if key is not found
    
    Throws:
    
    IOException
  - getObjectListingChunkForPath
```
protected ObjectUnderFileSystem.ObjectListingChunk getObjectListingChunkForPath(String path,
                                                                                boolean recursive)
                                                                         throws IOException
```
    Throws:
    
    IOException
  - getObjectListingChunkForPath
```
@Nullable
protected ObjectUnderFileSystem.ObjectListingChunk getObjectListingChunkForPath(String path,
                                                                                          boolean recursive,
                                                                                          String startAfter,
                                                                                          int batchSize)
                                                                                   throws IOException
```
    Gets a (partial) object listing for the given path.
    
    Parameters:
    
    path - of pseudo-directory
    
    recursive - whether to request immediate children only, or all descendants
    
    Returns:
    
    chunked object listing, or null if the path does not exist as a pseudo-directory
    
    Throws:
    
    IOException
  - getRootKey
```
protected abstract String getRootKey()
```
    Get full path of root in object store.
    
    Returns:
    
    full path including scheme and bucket
  - listInternal
```
@Nullable
protected UfsStatus[] listInternal(String path,
                                             ListOptions options)
                                      throws IOException
```
    Lists the files in the given path, the paths will be their logical names and not contain the folder suffix. Note that, the list results are unsorted.
    
    Parameters:
    
    path - the key to list
    
    options - for listing
    
    Returns:
    
    an array of the file and folder names in this directory
    
    Throws:
    
    IOException
  - mkdirsInternal
```
protected boolean mkdirsInternal(String key)
```
    Creates a directory flagged file with the key and folder suffix.
    
    Parameters:
    
    key - the key to create a folder
    
    Returns:
    
    true if the operation was successful, false otherwise
  - openObject
```
protected abstract InputStream openObject(String key,
                                          OpenOptions options,
                                          RetryPolicy retryPolicy)
                                   throws IOException
```
    Internal function to open an input stream to an object.
    
    Parameters:
    
    key - the key to open
    
    options - the open options
    
    retryPolicy - the retry policy of the opened stream to solve eventual consistency issue
    
    Returns:
    
    an InputStream to read from key
    
    Throws:
    
    IOException
  - parentExists
```
protected boolean parentExists(String path)
                        throws IOException
```
    Treating the object store as a file system, checks if the parent directory exists.
    
    Parameters:
    
    path - the path to check
    
    Returns:
    
    true if the parent exists or if the path is root, false otherwise
    
    Throws:
    
    IOException
  - stripPrefixIfPresent
```
public String stripPrefixIfPresent(String path)
```
    Strips the bucket prefix or the preceding path separator from the path if it is present. For example, for input path ufs://my-bucket-name/my-path/file, the output would be my-path/file. If path is an absolute path like /my-path/file, the output would be my-path/file. This method will leave keys without a prefix unaltered, ie. my-path/file returns my-path/file.
    
    Parameters:
    
    path - the path to strip
    
    Returns:
    
    the path without the bucket prefix
  - retryOnException
```
protected <T> T retryOnException(ObjectUnderFileSystem.ObjectStoreOperation<T> op,
                                 java.util.function.Supplier<String> description)
                          throws IOException
```
    Retries the given object store operation when it throws exceptions to resolve eventual consistency issue.
    
    Parameters:
    
    op - the object store operation to retry
    
    description - the description regarding the operation
    
    Returns:
    
    the operation result if operation succeed
    
    Throws:
    
    IOException

Class ObjectUnderFileSystem

Nested Class Summary

Nested classes/interfaces inherited from interface alluxio.underfs.UnderFileSystem

Field Summary

Fields inherited from class alluxio.underfs.BaseUnderFileSystem

Constructor Summary

Method Summary

Methods inherited from class alluxio.underfs.BaseUnderFileSystem

Methods inherited from class java.lang.Object

Methods inherited from interface alluxio.underfs.UnderFileSystem

Field Detail

PATH_SEPARATOR_CHAR

PATH_SEPARATOR

mExecutorService

mRootKeySupplier

Constructor Detail

ObjectUnderFileSystem

Method Detail

cleanup

close

connectFromMaster

connectFromWorker

create

createNonexistingFile

createNonexistingFile

deleteFile

deleteExistingFile

deleteDirectory

deleteExistingDirectory

deleteExistingDirectory

getBlockSizeByte

getDirectoryStatus

getExistingDirectoryStatus

getFileLocations

getFileLocations

getSpace

getFileStatus

getExistingFileStatus

getStatus

getExistingStatus

isDirectory

isExistingDirectory

isFile

isObjectStorage

listStatus

listStatus

listStatusIterable

mkdirs

open

openExistingFile

openExistingFile

renameDirectory

renameRenamableDirectory

renameFile

renameRenamableFile

supportsFlush

createEmptyObject

createObject

convertToFolderName

copyObject

deleteObject

deleteObjects

getPermissions

getListingChunkLengthMax

getListingChunkLength

getObjectStatus

getParentPath

isRoot

getChildName

getFolderSuffix

getObjectListingChunk

getObjectListingChunk

getObjectListingChunkForPath

getObjectListingChunkForPath

getRootKey

listInternal

mkdirsInternal

openObject

parentExists

stripPrefixIfPresent