public class ManagedBlockingUfsForwarder extends Object implements UnderFileSystem
UnderFileSystem
objects that works through with ForkJoinPool's
managed blocking.
If UFS calls are being done on a ForkJoinWorkerThread
, then
this forwarder will make sure UFS operations are treated as blocking operations
for compensating the ForkJoinPool.UnderFileSystem.Factory, UnderFileSystem.SpaceType
Constructor and Description |
---|
ManagedBlockingUfsForwarder(UnderFileSystem ufs)
Creates
ManagedBlockingUfsForwarder instance. |
Modifier and Type | Method and Description |
---|---|
void |
cleanup()
Cleans up the under file system.
|
void |
close() |
void |
connectFromMaster(String hostname)
Takes any necessary actions required to establish a connection to the under file system from
the given master host e.g.
|
void |
connectFromWorker(String hostname)
Takes any necessary actions required to establish a connection to the under file system from
the given worker host e.g.
|
OutputStream |
create(String path)
Creates a file in the under file system with the indicated name.
|
OutputStream |
create(String path,
CreateOptions options)
Creates a file in the under file system with the specified
CreateOptions . |
OutputStream |
createNonexistingFile(String path)
Creates a file in the under file system with the indicated name.
|
OutputStream |
createNonexistingFile(String path,
CreateOptions options)
Creates a file in the under file system with the specified
CreateOptions . |
boolean |
deleteDirectory(String path)
Deletes a directory from the under file system with the indicated name non-recursively.
|
boolean |
deleteDirectory(String path,
DeleteOptions options)
Deletes a directory from the under file system with the indicated name.
|
boolean |
deleteExistingDirectory(String path)
Deletes a directory from the under file system.
|
boolean |
deleteExistingDirectory(String path,
DeleteOptions options)
Deletes a directory from the under file system with the indicated name.
|
boolean |
deleteExistingFile(String path)
Deletes a file from the under file system with the indicated name.
|
boolean |
deleteFile(String path)
Deletes a file from the under file system with the indicated name.
|
boolean |
exists(String path)
Checks if a file or directory exists in under file system.
|
Pair<AccessControlList,DefaultAccessControlList> |
getAclPair(String path)
Gets the ACL and the Default ACL of a file or directory in under file system.
|
long |
getBlockSizeByte(String path)
Gets the block size of a file in under file system, in bytes.
|
AlluxioConfiguration |
getConfiguration()
Gets the under file system configuration.
|
UfsDirectoryStatus |
getDirectoryStatus(String path)
Gets the directory status.
|
UfsDirectoryStatus |
getExistingDirectoryStatus(String path)
Gets the directory status.
|
UfsFileStatus |
getExistingFileStatus(String path)
Gets the file status.
|
UfsStatus |
getExistingStatus(String path)
Gets the file or directory status.
|
List<String> |
getFileLocations(String path)
Gets the list of locations of the indicated path.
|
List<String> |
getFileLocations(String path,
FileLocationOptions options)
Gets the list of locations of the indicated path given options.
|
UfsFileStatus |
getFileStatus(String path,
GetStatusOptions options)
Gets the file status.
|
String |
getFingerprint(String path)
Computes and returns a fingerprint for the path.
|
UfsMode |
getOperationMode(Map<String,UfsMode> physicalUfsState)
An
UnderFileSystem may be composed of one or more "physical UFS"s. |
Fingerprint |
getParsedFingerprint(String path)
Computes and returns a fingerprint for the path.
|
Fingerprint |
getParsedFingerprint(String path,
String contentHash)
Same as
UnderFileSystem.getParsedFingerprint(String) except, will use the given content hash
as the Fingerprint.Tag.CONTENT_HASH field of the fingerprint
if non-null. |
List<String> |
getPhysicalStores()
An
UnderFileSystem may be composed of one or more "physical UFS"s. |
long |
getSpace(String path,
UnderFileSystem.SpaceType type)
Queries the under file system about the space of the indicated path (e.g., space left, space
used and etc).
|
UfsStatus |
getStatus(String path,
GetStatusOptions options)
Gets the file or directory status.
|
String |
getUnderFSType()
Returns the name of the under filesystem implementation.
|
boolean |
isDirectory(String path)
Checks if a directory exists in under file system.
|
boolean |
isExistingDirectory(String path)
Checks if a directory exists in under file system.
|
boolean |
isFile(String path)
Checks if a file exists in under file system.
|
boolean |
isObjectStorage() |
boolean |
isSeekable()
Denotes if the under storage supports seeking.
|
UfsStatus[] |
listStatus(String path)
Returns an array of statuses of the files and directories in the directory denoted by this
abstract pathname.
|
UfsStatus[] |
listStatus(String path,
ListOptions options)
Returns an array of statuses of the files and directories in the directory denoted by this
abstract pathname, with options.
|
Iterator<UfsStatus> |
listStatusIterable(String path,
ListOptions options,
String startAfter,
int batchSize)
Lists the ufs statuses iteratively.
|
boolean |
mkdirs(String path)
Creates the directory named by this abstract pathname.
|
boolean |
mkdirs(String path,
MkdirsOptions options)
Creates the directory named by this abstract pathname, with specified
MkdirsOptions . |
InputStream |
open(String path)
Opens an
InputStream for a file in under filesystem at the indicated path. |
InputStream |
open(String path,
OpenOptions options)
Opens an
InputStream for a file in under filesystem at the indicated path. |
InputStream |
openExistingFile(String path)
Opens an
InputStream for a file in under filesystem at the indicated path. |
InputStream |
openExistingFile(String path,
OpenOptions options)
Opens an
InputStream for a file in under filesystem at the indicated path. |
PositionReader |
openPositionRead(String path,
long fileLength)
Opens a file for position read.
|
void |
performListingAsync(String path,
String continuationToken,
String startAfter,
DescendantType descendantType,
boolean checkStatus,
java.util.function.Consumer<UfsLoadResult> onComplete,
java.util.function.Consumer<Throwable> onError)
Lists the ufs statuses for a given path.
|
boolean |
renameDirectory(String src,
String dst)
Renames a directory from
src to dst in under file system. |
boolean |
renameFile(String src,
String dst)
Renames a file from
src to dst in under file system. |
boolean |
renameRenamableDirectory(String src,
String dst)
Renames a directory from
src to dst in under file system. |
boolean |
renameRenamableFile(String src,
String dst)
Renames a file from
src to dst in under file system. |
AlluxioURI |
resolveUri(AlluxioURI ufsBaseUri,
String alluxioPath)
Returns an
AlluxioURI representation for the UnderFileSystem given a base
UFS URI, and the Alluxio path from the base. |
void |
setAclEntries(String path,
List<AclEntry> aclEntries)
Sets the access control list of a file or directory in under file system.
|
void |
setMode(String path,
short mode)
Changes posix file mode.
|
void |
setOwner(String path,
String owner,
String group)
Sets the user and group of the given path.
|
boolean |
supportsFlush()
Whether this type of UFS supports flush.
|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
getFileStatus, getStatus, getTemporaryToken, isTemporaryConnectionToken, listStatuses, refreshTemporaryToken, setTemporaryTokenRefreshContext
public ManagedBlockingUfsForwarder(UnderFileSystem ufs)
ManagedBlockingUfsForwarder
instance.ufs
- the underlying ufspublic void cleanup() throws IOException
UnderFileSystem
cleanup
in interface UnderFileSystem
IOException
public void connectFromMaster(String hostname) throws IOException
UnderFileSystem
Depending on the implementation this may be a no-op
connectFromMaster
in interface UnderFileSystem
hostname
- the host that wants to connect to the under file systemIOException
public void connectFromWorker(String hostname) throws IOException
UnderFileSystem
Depending on the implementation this may be a no-op
connectFromWorker
in interface UnderFileSystem
hostname
- the host that wants to connect to the under file systemIOException
public OutputStream create(String path) throws IOException
UnderFileSystem
create
in interface UnderFileSystem
path
- the file nameOutputStream
objectIOException
public OutputStream create(String path, CreateOptions options) throws IOException
UnderFileSystem
CreateOptions
.
Implementations should make sure that the path under creation appears in listings only
after a successful close and that contents are written in its entirety or not at all.create
in interface UnderFileSystem
path
- the file nameoptions
- the options for createOutputStream
objectIOException
public OutputStream createNonexistingFile(String path) throws IOException
UnderFileSystem
UnderFileSystem.create(String)
but
deals with the delete-then-create eventual consistency issue.createNonexistingFile
in interface UnderFileSystem
path
- the file nameOutputStream
objectIOException
public OutputStream createNonexistingFile(String path, CreateOptions options) throws IOException
UnderFileSystem
CreateOptions
.
Similar to UnderFileSystem.create(String, CreateOptions)
but
deals with the delete-then-create eventual consistency issue.createNonexistingFile
in interface UnderFileSystem
path
- the file nameoptions
- the options for createOutputStream
objectIOException
public boolean deleteDirectory(String path) throws IOException
UnderFileSystem
deleteDirectory
in interface UnderFileSystem
path
- of the directory to deleteIOException
public boolean deleteDirectory(String path, DeleteOptions options) throws IOException
UnderFileSystem
deleteDirectory
in interface UnderFileSystem
path
- of the directory to deleteoptions
- for directory delete semanticsIOException
public boolean deleteExistingDirectory(String path) throws IOException
UnderFileSystem
UnderFileSystem.deleteDirectory(String)
but
deals with the create-delete eventual consistency issue.deleteExistingDirectory
in interface UnderFileSystem
path
- of the directory to deleteIOException
public boolean deleteExistingDirectory(String path, DeleteOptions options) throws IOException
UnderFileSystem
UnderFileSystem.deleteDirectory(String, DeleteOptions)
but
deals with the create-then-delete eventual consistency issue.deleteExistingDirectory
in interface UnderFileSystem
path
- of the directory to deleteoptions
- for directory delete semanticsIOException
public boolean deleteFile(String path) throws IOException
UnderFileSystem
deleteFile
in interface UnderFileSystem
path
- of the file to deleteIOException
public boolean deleteExistingFile(String path) throws IOException
UnderFileSystem
UnderFileSystem.deleteFile(String)
but
deals with the create-then-delete eventual consistency issue.deleteExistingFile
in interface UnderFileSystem
path
- of the file to deleteIOException
public boolean exists(String path) throws IOException
UnderFileSystem
exists
in interface UnderFileSystem
path
- the absolute pathIOException
public Pair<AccessControlList,DefaultAccessControlList> getAclPair(String path) throws IOException
UnderFileSystem
getAclPair
in interface UnderFileSystem
path
- the path to the file or directoryIOException
- if ACL is supported and enabled but cannot be retrievedpublic long getBlockSizeByte(String path) throws IOException
UnderFileSystem
getBlockSizeByte
in interface UnderFileSystem
path
- the file nameIOException
public AlluxioConfiguration getConfiguration() throws IOException
UnderFileSystem
getConfiguration
in interface UnderFileSystem
IOException
public UfsDirectoryStatus getDirectoryStatus(String path) throws IOException
UnderFileSystem
getDirectoryStatus
in interface UnderFileSystem
path
- the path to the directoryFileNotFoundException
- when the path does not existIOException
public UfsDirectoryStatus getExistingDirectoryStatus(String path) throws IOException
UnderFileSystem
UnderFileSystem.getDirectoryStatus(String)
but
deals with the write-then-get-status eventual consistency issue.getExistingDirectoryStatus
in interface UnderFileSystem
path
- the path to the directoryIOException
public List<String> getFileLocations(String path) throws IOException
UnderFileSystem
getFileLocations
in interface UnderFileSystem
path
- the file nameIOException
public List<String> getFileLocations(String path, FileLocationOptions options) throws IOException
UnderFileSystem
getFileLocations
in interface UnderFileSystem
path
- the file nameoptions
- method optionsIOException
public UfsFileStatus getFileStatus(String path, GetStatusOptions options) throws IOException
UnderFileSystem
getFileStatus
in interface UnderFileSystem
path
- the path to the fileoptions
- method optionsFileNotFoundException
- when the path does not existIOException
public UfsFileStatus getExistingFileStatus(String path) throws IOException
UnderFileSystem
UnderFileSystem.getFileStatus(String)
but
deals with the write-then-get-status eventual consistency issue.getExistingFileStatus
in interface UnderFileSystem
path
- the path to the fileIOException
public String getFingerprint(String path)
UnderFileSystem
Constants.INVALID_UFS_FINGERPRINT
if there is any error.getFingerprint
in interface UnderFileSystem
path
- the path to compute the fingerprint forpublic Fingerprint getParsedFingerprint(String path)
UnderFileSystem
Fingerprint.INVALID_FINGERPRINT
if there is any error.
The default implementation relies on UnderFileSystem.getFingerprint(String)
and there is one extra
parsing. This default implementation is mainly for backward compatibility.
Override this for performance.getParsedFingerprint
in interface UnderFileSystem
path
- the path to compute the fingerprint forpublic Fingerprint getParsedFingerprint(String path, @Nullable String contentHash)
UnderFileSystem
UnderFileSystem.getParsedFingerprint(String)
except, will use the given content hash
as the Fingerprint.Tag.CONTENT_HASH
field of the fingerprint
if non-null. This is intended to be used when the file is already in Alluxio and
a fingerprint is being created based on that file where the content hash has already
been computed.getParsedFingerprint
in interface UnderFileSystem
path
- the path to compute the fingerprint forcontentHash
- is used as the Fingerprint.Tag.CONTENT_HASH
field when creating the fingerprint.public UfsMode getOperationMode(Map<String,UfsMode> physicalUfsState)
UnderFileSystem
UnderFileSystem
may be composed of one or more "physical UFS"s. This method is used
to determine the operation mode based on the physical UFS operation modes. For example, if this
UnderFileSystem
is composed of physical UFS hdfs://ns1/ and hdfs://ns2/ with read
operations split b/w the two, with physicalUfsState{hdfs://ns1/:NO_ACCESS,
hdfs://ns2/:READ_WRITE} this method can return READ_ONLY to allow reads to proceed from
hdfs://ns2/.getOperationMode
in interface UnderFileSystem
physicalUfsState
- the state of physical UFSs for this UnderFileSystem
; keys are
expected to be normalized (ending with /)public List<String> getPhysicalStores()
UnderFileSystem
UnderFileSystem
may be composed of one or more "physical UFS"s. This method
returns all underlying physical stores; normalized with only scheme and authority.getPhysicalStores
in interface UnderFileSystem
UnderFileSystem
is composed ofpublic long getSpace(String path, UnderFileSystem.SpaceType type) throws IOException
UnderFileSystem
getSpace
in interface UnderFileSystem
path
- the path to querytype
- the type of queriesIOException
public UfsStatus getStatus(String path, GetStatusOptions options) throws IOException
UnderFileSystem
getStatus
in interface UnderFileSystem
path
- the path to get the statusoptions
- method optionsFileNotFoundException
- when the path does not existIOException
public UfsStatus getExistingStatus(String path) throws IOException
UnderFileSystem
UnderFileSystem.getStatus(String)
but
deals with the write-then-get-status eventual consistency issue.getExistingStatus
in interface UnderFileSystem
path
- the path to get the statusIOException
public String getUnderFSType()
UnderFileSystem
getUnderFSType
in interface UnderFileSystem
public boolean isDirectory(String path) throws IOException
UnderFileSystem
isDirectory
in interface UnderFileSystem
path
- the absolute directory pathIOException
public boolean isExistingDirectory(String path) throws IOException
UnderFileSystem
UnderFileSystem.isDirectory(String)
but
deals with the write-then-list eventual consistency issue.isExistingDirectory
in interface UnderFileSystem
path
- the absolute directory pathIOException
public boolean isFile(String path) throws IOException
UnderFileSystem
isFile
in interface UnderFileSystem
path
- the absolute file pathIOException
public boolean isObjectStorage()
isObjectStorage
in interface UnderFileSystem
public boolean isSeekable()
UnderFileSystem
SeekableUnderFileInputStream
in the UnderFileSystem.open(String, OpenOptions)
method.isSeekable
in interface UnderFileSystem
public UfsStatus[] listStatus(String path) throws IOException
UnderFileSystem
If this abstract pathname does not denote a directory, then this method returns null
.
Otherwise an array of statuses is returned, one for each file or directory in the directory.
Names denoting the directory itself and the directory's parent directory are not included in
the result. Each string is a file name rather than a complete path.
There is no guarantee that the name strings in the resulting array will appear in any specific order; they are not, in particular, guaranteed to appear in alphabetical order.
listStatus
in interface UnderFileSystem
path
- the abstract pathname to listnull
if this abstract pathname does not denote a directory.IOException
public UfsStatus[] listStatus(String path, ListOptions options) throws IOException
UnderFileSystem
If this abstract pathname does not denote a directory, then this method returns null
.
Otherwise an array of statuses is returned, one for each file or directory. Names denoting the
directory itself and the directory's parent directory are not included in the result. Each
string is a path relative to the given directory.
There is no guarantee that the name strings in the resulting array will appear in any specific order; they are not, in particular, guaranteed to appear in alphabetical order.
listStatus
in interface UnderFileSystem
path
- the abstract pathname to listoptions
- for list directorynull
if this abstract pathname does not denote a directory.IOException
public boolean mkdirs(String path) throws IOException
UnderFileSystem
mkdirs
in interface UnderFileSystem
path
- the folder to createtrue
if and only if the directory was created; false
otherwiseIOException
public boolean mkdirs(String path, MkdirsOptions options) throws IOException
UnderFileSystem
MkdirsOptions
. If the folder already exists, the method returns false.mkdirs
in interface UnderFileSystem
path
- the folder to createoptions
- the options for mkdirstrue
if and only if the directory was created; false
otherwiseIOException
public InputStream open(String path) throws IOException
UnderFileSystem
InputStream
for a file in under filesystem at the indicated path.open
in interface UnderFileSystem
path
- the file nameInputStream
objectIOException
public InputStream open(String path, OpenOptions options) throws IOException
UnderFileSystem
InputStream
for a file in under filesystem at the indicated path.open
in interface UnderFileSystem
path
- the file nameoptions
- to open input streamInputStream
objectIOException
public InputStream openExistingFile(String path) throws IOException
UnderFileSystem
InputStream
for a file in under filesystem at the indicated path.
Similar to UnderFileSystem.open(String)
but
deals with the write-then-read eventual consistency issue.openExistingFile
in interface UnderFileSystem
path
- the file nameInputStream
objectIOException
public InputStream openExistingFile(String path, OpenOptions options) throws IOException
UnderFileSystem
InputStream
for a file in under filesystem at the indicated path.
Similar to UnderFileSystem.open(String, OpenOptions)
but
deals with the write-then-read eventual consistency issue.openExistingFile
in interface UnderFileSystem
path
- the file nameoptions
- to open input streamInputStream
objectIOException
public PositionReader openPositionRead(String path, long fileLength)
UnderFileSystem
openPositionRead
in interface UnderFileSystem
path
- the path to readfileLength
- the file lengthpublic boolean renameDirectory(String src, String dst) throws IOException
UnderFileSystem
src
to dst
in under file system.renameDirectory
in interface UnderFileSystem
src
- the source directory pathdst
- the destination directory pathIOException
public boolean renameRenamableDirectory(String src, String dst) throws IOException
UnderFileSystem
src
to dst
in under file system.
Similar to UnderFileSystem.renameDirectory(String, String)
but
deals with the write-src-then-rename and delete-dst-then-rename eventual consistency issue.renameRenamableDirectory
in interface UnderFileSystem
src
- the source directory pathdst
- the destination directory pathIOException
public boolean renameFile(String src, String dst) throws IOException
UnderFileSystem
src
to dst
in under file system.renameFile
in interface UnderFileSystem
src
- the source file pathdst
- the destination file pathIOException
public boolean renameRenamableFile(String src, String dst) throws IOException
UnderFileSystem
src
to dst
in under file system.
Similar to UnderFileSystem.renameFile(String, String)
but
deals with the write-src-then-rename and delete-dst-then-rename eventual consistency issue.renameRenamableFile
in interface UnderFileSystem
src
- the source file pathdst
- the destination file pathIOException
public AlluxioURI resolveUri(AlluxioURI ufsBaseUri, String alluxioPath)
UnderFileSystem
AlluxioURI
representation for the UnderFileSystem
given a base
UFS URI, and the Alluxio path from the base.
The default implementation simply concatenates the path to the base URI. This should be
overridden if a subclass needs alternate functionality.resolveUri
in interface UnderFileSystem
ufsBaseUri
- the base AlluxioURI
in the ufsalluxioPath
- the path in Alluxio from the given baseAlluxioURI
representing the Alluxio pathpublic void setAclEntries(String path, List<AclEntry> aclEntries) throws IOException
UnderFileSystem
setAclEntries
in interface UnderFileSystem
path
- the path to the file or directoryaclEntries
- the access control list + default acl represented in a list of acl entriesIOException
public void setMode(String path, short mode) throws IOException
UnderFileSystem
setMode
in interface UnderFileSystem
path
- the path of the filemode
- the mode to set in short format, e.g. 0777IOException
public void setOwner(String path, String owner, String group) throws IOException
UnderFileSystem
setOwner
in interface UnderFileSystem
path
- the path of the fileowner
- the new owner to set, unchanged if nullgroup
- the new group to set, unchanged if nullIOException
public boolean supportsFlush() throws IOException
UnderFileSystem
supportsFlush
in interface UnderFileSystem
IOException
public void close() throws IOException
close
in interface Closeable
close
in interface AutoCloseable
IOException
public Iterator<UfsStatus> listStatusIterable(String path, ListOptions options, String startAfter, int batchSize) throws IOException
UnderFileSystem
listStatusIterable
in interface UnderFileSystem
path
- the abstract pathname to listoptions
- for list directorystartAfter
- the start after tokenbatchSize
- the batch sizenull
if this abstract pathname does not denote a directory.IOException
public void performListingAsync(String path, @Nullable String continuationToken, @Nullable String startAfter, DescendantType descendantType, boolean checkStatus, java.util.function.Consumer<UfsLoadResult> onComplete, java.util.function.Consumer<Throwable> onError)
AsyncUfsClient
UfsStatus.getName()
function for the returned values should include the full path of each
item from the UFS root (not including the bucket name for object stores).
It differs from a traditional listing in that if the input variable
checkStatus is true, the UfsStatus
for the base path should
be included at the start of the results. The function should return
immediately, and perform the operation asynchronously.performListingAsync
in interface AsyncUfsClient
path
- the path in ufscontinuationToken
- the continuation tokenstartAfter
- the start after string where the loading starts fromdescendantType
- the load descendant type (NONE/ONE/ALL)checkStatus
- if true the call will perform a GetStatus on the path
to see if an object exists, which should be returned
as part of the resultonComplete
- the callback when the load is completeonError
- the callback when the load encountered an errorCopyright © 2023. All Rights Reserved.