alluxio.underfs.s3a

Class S3AUnderFileSystem

java.lang.Object

alluxio.underfs.BaseUnderFileSystem

alluxio.underfs.ObjectUnderFileSystem

alluxio.underfs.s3a.S3AUnderFileSystem

All Implemented Interfaces:

AsyncUfsClient, UnderFileSystem, Closeable, AutoCloseable

@ThreadSafe
public class S3AUnderFileSystem
extends ObjectUnderFileSystem

S3 UnderFileSystem implementation based on the aws-java-sdk-s3 library.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected class	S3AUnderFileSystem.S3AObjectListingChunk Wrapper over S3 ListObjectsV2Request.
protected class	S3AUnderFileSystem.S3AObjectListingChunkV1 Wrapper over S3 ListObjectsRequest.

Nested classes/interfaces inherited from class alluxio.underfs.ObjectUnderFileSystem

ObjectUnderFileSystem.DeleteBuffer, ObjectUnderFileSystem.ObjectListingChunk, ObjectUnderFileSystem.ObjectPermissions, ObjectUnderFileSystem.ObjectStatus, ObjectUnderFileSystem.ObjectStoreOperation<T>, ObjectUnderFileSystem.OperationBuffer<T>, ObjectUnderFileSystem.RenameBuffer, ObjectUnderFileSystem.UfsStatusIterator

Nested classes/interfaces inherited from interface alluxio.underfs.UnderFileSystem

UnderFileSystem.Factory, UnderFileSystem.SpaceType

Field Summary

Fields

Modifier and Type	Field and Description
protected static String	DIR_HASH Static hash for a directory's empty contents.
protected static long	MULTIPART_COPY_THRESHOLD Threshold to do multipart copy.

Fields inherited from class alluxio.underfs.ObjectUnderFileSystem

mExecutorService, mRootKeySupplier, PATH_SEPARATOR, PATH_SEPARATOR_CHAR

Fields inherited from class alluxio.underfs.BaseUnderFileSystem

EMPTY_ACL, mUfsConf, mUri

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	S3AUnderFileSystem(AlluxioURI uri, com.amazonaws.services.s3.AmazonS3 amazonS3Client, software.amazon.awssdk.services.s3.S3AsyncClient asyncClient, String bucketName, ExecutorService executor, com.amazonaws.services.s3.transfer.TransferManager transferManager, UnderFileSystemConfiguration conf, boolean streamingUploadEnabled, boolean multipartUploadEnabled) Constructor for S3AUnderFileSystem.

Method Summary

Modifier and Type	Method and Description
void	cleanup() Cleans up the under file system.
void	close()
protected boolean	copyObject(String src, String dst) Copies an object to another key.
static com.amazonaws.services.s3.AmazonS3	createAmazonS3(com.amazonaws.auth.AWSCredentialsProvider credentialsProvider, com.amazonaws.ClientConfiguration clientConf, com.amazonaws.client.builder.AwsClientBuilder.EndpointConfiguration endpointConfiguration, UnderFileSystemConfiguration conf) Create an AmazonS3 client.
static software.amazon.awssdk.services.s3.S3AsyncClient	createAmazonS3Async(UnderFileSystemConfiguration conf, com.amazonaws.ClientConfiguration clientConf) Create an async S3 client.
static com.amazonaws.auth.AWSCredentialsProvider	createAwsCredentialsProvider(UnderFileSystemConfiguration conf)
boolean	createEmptyObject(String key) Creates a zero-byte object used to encode a directory.
protected static com.amazonaws.client.builder.AwsClientBuilder.EndpointConfiguration	createEndpointConfiguration(UnderFileSystemConfiguration conf, com.amazonaws.ClientConfiguration clientConf) Creates an endpoint configuration.
static S3AUnderFileSystem	createInstance(AlluxioURI uri, UnderFileSystemConfiguration conf) Constructs a new instance of S3AUnderFileSystem.
protected OutputStream	createObject(String key) Creates an OutputStream for object uploads.
protected boolean	deleteObject(String key) Internal function to delete a key.
protected List<String>	deleteObjects(List<String> keys) Internal function to delete a list of keys.
protected String	getBucketName()
protected com.amazonaws.services.s3.AmazonS3	getClient()
protected com.google.common.util.concurrent.ListeningExecutorService	getExecutor()
protected String	getFolderSuffix() Get suffix used to encode a directory.
protected boolean	getMultipartUploadEnabled()
protected ObjectUnderFileSystem.ObjectListingChunk	getObjectListingChunk(String key, boolean recursive) Gets a (partial) object listing result for the given key.
protected ObjectUnderFileSystem.ObjectListingChunk	getObjectListingChunk(String key, boolean recursive, String startAfter, int batchSize) Gets a (partial) object listing result for the given key.
protected ObjectUnderFileSystem.ObjectStatus	getObjectStatus(String key) Get metadata information about object.
protected ObjectUnderFileSystem.ObjectPermissions	getPermissions() Permissions for the mounted bucket.
protected String	getRootKey() Get full path of root in object store.
protected boolean	getStreamingUploadEnabled()
protected com.amazonaws.services.s3.transfer.TransferManager	getTransferManager()
String	getUnderFSType() Returns the name of the under filesystem implementation.
protected InputStream	openObject(String key, OpenOptions options, RetryPolicy retryPolicy) Internal function to open an input stream to an object.
S3APositionReader	openPositionRead(String path, long fileLength) Opens a file for position read.
void	performListingAsync(String path, String continuationToken, String startAfter, DescendantType descendantType, boolean checkStatus, java.util.function.Consumer<UfsLoadResult> onComplete, java.util.function.Consumer<Throwable> onError) Lists the ufs statuses for a given path.
protected void	setClient(com.amazonaws.services.s3.AmazonS3 client)
void	setMode(String path, short mode) Changes posix file mode.
void	setOwner(String path, String user, String group) Sets the user and group of the given path.

Methods inherited from class alluxio.underfs.ObjectUnderFileSystem

connectFromMaster, connectFromWorker, convertToFolderName, create, createNonexistingFile, createNonexistingFile, deleteDirectory, deleteExistingDirectory, deleteExistingDirectory, deleteExistingFile, deleteFile, getBlockSizeByte, getChildName, getDirectoryStatus, getExistingDirectoryStatus, getExistingFileStatus, getExistingStatus, getFileLocations, getFileLocations, getFileStatus, getListingChunkLength, getListingChunkLengthMax, getObjectListingChunkForPath, getObjectListingChunkForPath, getParentPath, getSpace, getStatus, isDirectory, isExistingDirectory, isFile, isObjectStorage, isRoot, listInternal, listStatus, listStatus, listStatusIterable, mkdirs, mkdirsInternal, open, openExistingFile, openExistingFile, parentExists, renameDirectory, renameFile, renameRenamableDirectory, renameRenamableFile, retryOnException, stripPrefixIfPresent, supportsFlush

Methods inherited from class alluxio.underfs.BaseUnderFileSystem

create, deleteDirectory, exists, getAclPair, getConfiguration, getFingerprint, getOperationMode, getParsedFingerprint, getParsedFingerprint, getPhysicalStores, isSeekable, mkdirs, open, resolveUri, setAclEntries, validatePath

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface alluxio.underfs.UnderFileSystem

getFileStatus, getStatus, getTemporaryToken, isTemporaryConnectionToken, listStatuses, refreshTemporaryToken, setTemporaryTokenRefreshContext

Field Detail

DIR_HASH

protected static final String DIR_HASH

Static hash for a directory's empty contents.

MULTIPART_COPY_THRESHOLD

protected static final long MULTIPART_COPY_THRESHOLD

Threshold to do multipart copy.

See Also:

Constant Field Values

Constructor Detail

S3AUnderFileSystem

protected S3AUnderFileSystem(AlluxioURI uri,
                             com.amazonaws.services.s3.AmazonS3 amazonS3Client,
                             software.amazon.awssdk.services.s3.S3AsyncClient asyncClient,
                             String bucketName,
                             ExecutorService executor,
                             com.amazonaws.services.s3.transfer.TransferManager transferManager,
                             UnderFileSystemConfiguration conf,
                             boolean streamingUploadEnabled,
                             boolean multipartUploadEnabled)

Constructor for S3AUnderFileSystem.

Parameters:

uri - the AlluxioURI for this UFS

amazonS3Client - AWS-SDK S3 client

asyncClient - AWS S3 async client

bucketName - bucket name of user's configured Alluxio bucket

executor - the executor for executing upload tasks

transferManager - Transfer Manager for efficient I/O to S3

conf - configuration for this S3A ufs

streamingUploadEnabled - whether streaming upload is enabled

multipartUploadEnabled - whether multipart upload is enabled

Method Detail

createAwsCredentialsProvider

public static com.amazonaws.auth.AWSCredentialsProvider createAwsCredentialsProvider(UnderFileSystemConfiguration conf)

Parameters:

conf - the configuration for this UFS

Returns:

the created AWSCredentialsProvider instance

createInstance

public static S3AUnderFileSystem createInstance(AlluxioURI uri,
                                                UnderFileSystemConfiguration conf)

Constructs a new instance of S3AUnderFileSystem.

Parameters:

uri - the AlluxioURI for this UFS

conf - the configuration for this UFS

Returns:

the created S3AUnderFileSystem instance

createAmazonS3Async

public static software.amazon.awssdk.services.s3.S3AsyncClient createAmazonS3Async(UnderFileSystemConfiguration conf,
                                                                                   com.amazonaws.ClientConfiguration clientConf)

Create an async S3 client.

Parameters:

conf - the conf

clientConf - the client conf

Returns:

the client

createAmazonS3

public static com.amazonaws.services.s3.AmazonS3 createAmazonS3(com.amazonaws.auth.AWSCredentialsProvider credentialsProvider,
                                                                com.amazonaws.ClientConfiguration clientConf,
                                                                com.amazonaws.client.builder.AwsClientBuilder.EndpointConfiguration endpointConfiguration,
                                                                UnderFileSystemConfiguration conf)

Create an AmazonS3 client.

Parameters:

credentialsProvider - the credential provider

clientConf - the client config

endpointConfiguration - the endpoint config

conf - the Ufs config

Returns:

the AmazonS3 client

createEndpointConfiguration

@Nullable
protected static com.amazonaws.client.builder.AwsClientBuilder.EndpointConfiguration createEndpointConfiguration(UnderFileSystemConfiguration conf,
                                                                                                                           com.amazonaws.ClientConfiguration clientConf)

Creates an endpoint configuration.

Parameters:

conf - the alluxio conf

clientConf - the aws conf

Returns:

the endpoint configuration

openPositionRead

public S3APositionReader openPositionRead(String path,
                                          long fileLength)

Description copied from interface: UnderFileSystem

Opens a file for position read.

Parameters:

path - the path to read

fileLength - the file length

Returns:

the position reader

getUnderFSType

public String getUnderFSType()

Description copied from interface: UnderFileSystem

Returns the name of the under filesystem implementation. The name should be lowercase and not include any spaces, e.g. "hdfs", "s3".

Returns:

name of the under filesystem implementation

setOwner

public void setOwner(String path,
                     String user,
                     String group)

Description copied from interface: UnderFileSystem

Sets the user and group of the given path. An empty implementation should be provided if unsupported.

Parameters:

path - the path of the file

user - the new owner to set, unchanged if null

group - the new group to set, unchanged if null

setMode

public void setMode(String path,
                    short mode)

Description copied from interface: UnderFileSystem

Changes posix file mode.

Parameters:

path - the path of the file

mode - the mode to set in short format, e.g. 0777

cleanup

public void cleanup()

Description copied from interface: UnderFileSystem

Cleans up the under file system. If any data or files are created and not completed/aborted correctly in normal ways, they should be cleaned in this method.

Specified by:

cleanup in interface UnderFileSystem

Overrides:

cleanup in class ObjectUnderFileSystem

close

public void close()

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Overrides:

close in class ObjectUnderFileSystem

copyObject

protected boolean copyObject(String src,
                             String dst)

Description copied from class: ObjectUnderFileSystem

Copies an object to another key.

Specified by:

copyObject in class ObjectUnderFileSystem

Parameters:

src - the source key to copy

dst - the destination key to copy to

Returns:

true if the operation was successful, false otherwise

getClient

protected com.amazonaws.services.s3.AmazonS3 getClient()

setClient

protected void setClient(com.amazonaws.services.s3.AmazonS3 client)

getBucketName

protected String getBucketName()

getExecutor

protected com.google.common.util.concurrent.ListeningExecutorService getExecutor()

getTransferManager

protected com.amazonaws.services.s3.transfer.TransferManager getTransferManager()

getStreamingUploadEnabled

protected boolean getStreamingUploadEnabled()

getMultipartUploadEnabled

protected boolean getMultipartUploadEnabled()

createEmptyObject

public boolean createEmptyObject(String key)

Description copied from class: ObjectUnderFileSystem

Creates a zero-byte object used to encode a directory.

Specified by:

createEmptyObject in class ObjectUnderFileSystem

Parameters:

key - the key to create

Returns:

true if the operation was successful

createObject

protected OutputStream createObject(String key)
                             throws IOException

Description copied from class: ObjectUnderFileSystem

Creates an OutputStream for object uploads.

Specified by:

createObject in class ObjectUnderFileSystem

Parameters:

key - ufs key including scheme and bucket

Returns:

new OutputStream

Throws:

IOException

deleteObject

protected boolean deleteObject(String key)

Description copied from class: ObjectUnderFileSystem

Internal function to delete a key.

Specified by:

deleteObject in class ObjectUnderFileSystem

Parameters:

key - the key to delete

Returns:

true if successful, false if an exception is thrown

deleteObjects

protected List<String> deleteObjects(List<String> keys)
                              throws IOException

Description copied from class: ObjectUnderFileSystem

Internal function to delete a list of keys.

Overrides:

deleteObjects in class ObjectUnderFileSystem

Parameters:

keys - the list of keys to delete

Returns:

list of successfully deleted keys

Throws:

IOException

getFolderSuffix

protected String getFolderSuffix()

Description copied from class: ObjectUnderFileSystem

Get suffix used to encode a directory.

Specified by:

getFolderSuffix in class ObjectUnderFileSystem

Returns:

folder suffix

getObjectListingChunk

@Nullable
protected ObjectUnderFileSystem.ObjectListingChunk getObjectListingChunk(String key,
                                                                                   boolean recursive)
                                                                            throws IOException

Description copied from class: ObjectUnderFileSystem

Gets a (partial) object listing result for the given key.

Specified by:

getObjectListingChunk in class ObjectUnderFileSystem

Parameters:

key - pseudo-directory key excluding header and bucket

recursive - whether to request immediate children only, or all descendants

Returns:

chunked object listing, or null if key is not found

Throws:

IOException

getObjectListingChunk

@Nullable
protected ObjectUnderFileSystem.ObjectListingChunk getObjectListingChunk(String key,
                                                                                   boolean recursive,
                                                                                   @Nullable
                                                                                   String startAfter,
                                                                                   int batchSize)
                                                                            throws IOException

Description copied from class: ObjectUnderFileSystem

Gets a (partial) object listing result for the given key.

Overrides:

getObjectListingChunk in class ObjectUnderFileSystem

Parameters:

key - pseudo-directory key excluding header and bucket

recursive - whether to request immediate children only, or all descendants

startAfter - indicates where the listing starts

batchSize - the batch size of each chunk

Returns:

chunked object listing, or null if key is not found

Throws:

IOException

performListingAsync

public void performListingAsync(String path,
                                @Nullable
                                String continuationToken,
                                @Nullable
                                String startAfter,
                                DescendantType descendantType,
                                boolean checkStatus,
                                java.util.function.Consumer<UfsLoadResult> onComplete,
                                java.util.function.Consumer<Throwable> onError)

Description copied from interface: AsyncUfsClient

Lists the ufs statuses for a given path. The UfsStatus.getName() function for the returned values should include the full path of each item from the UFS root (not including the bucket name for object stores). It differs from a traditional listing in that if the input variable checkStatus is true, the UfsStatus for the base path should be included at the start of the results. The function should return immediately, and perform the operation asynchronously.

Specified by:

performListingAsync in interface AsyncUfsClient

Overrides:

performListingAsync in class BaseUnderFileSystem

Parameters:

path - the path in ufs

continuationToken - the continuation token

startAfter - the start after string where the loading starts from

descendantType - the load descendant type (NONE/ONE/ALL)

checkStatus - if true the call will perform a GetStatus on the path to see if an object exists, which should be returned as part of the result

onComplete - the callback when the load is complete

onError - the callback when the load encountered an error

getObjectStatus

@Nullable
protected ObjectUnderFileSystem.ObjectStatus getObjectStatus(String key)

Description copied from class: ObjectUnderFileSystem

Get metadata information about object. Implementations should process the key as is, which may be a file or a directory key.

Specified by:

getObjectStatus in class ObjectUnderFileSystem

Parameters:

key - ufs key to get metadata for

Returns:

ObjectUnderFileSystem.ObjectStatus if key exists and successful, otherwise null

getPermissions

protected ObjectUnderFileSystem.ObjectPermissions getPermissions()

Description copied from class: ObjectUnderFileSystem

Permissions for the mounted bucket.

Specified by:

getPermissions in class ObjectUnderFileSystem

Returns:

permissions

getRootKey

protected String getRootKey()

Description copied from class: ObjectUnderFileSystem

Get full path of root in object store.

Specified by:

getRootKey in class ObjectUnderFileSystem

Returns:

full path including scheme and bucket

openObject

protected InputStream openObject(String key,
                                 OpenOptions options,
                                 RetryPolicy retryPolicy)

Description copied from class: ObjectUnderFileSystem

Internal function to open an input stream to an object.

Specified by:

openObject in class ObjectUnderFileSystem

Parameters:

key - the key to open

options - the open options

retryPolicy - the retry policy of the opened stream to solve eventual consistency issue

Returns:

an InputStream to read from key