alluxio.underfs.s3a

Class S3ALowLevelOutputStream

java.lang.Object

java.io.OutputStream

alluxio.underfs.ObjectLowLevelOutputStream

alluxio.underfs.s3a.S3ALowLevelOutputStream

All Implemented Interfaces:

ContentHashable, Closeable, Flushable, AutoCloseable

@NotThreadSafe
public class S3ALowLevelOutputStream
extends ObjectLowLevelOutputStream

Object storage low output stream for aws s3.

Field Summary

Fields

Modifier and Type	Field and Description
protected com.amazonaws.services.s3.AmazonS3	mClient The Amazon S3 client to interact with S3.
protected String	mUploadId The upload id of this multipart upload.

Fields inherited from class alluxio.underfs.ObjectLowLevelOutputStream

mBucketName, mClosed, mFile, mHash, mKey, mLocalOutputStream, mPartitionOffset, mPartitionSize, mRetryPolicy, mSingleCharWrite, mTmpDirs, UPLOAD_THRESHOLD

Constructor Summary

Constructors

Constructor and Description
S3ALowLevelOutputStream(String bucketName, String key, com.amazonaws.services.s3.AmazonS3 s3Client, com.google.common.util.concurrent.ListeningExecutorService executor, AlluxioConfiguration ufsConf) Constructs a new stream for writing a file.

Method Summary

Modifier and Type	Method and Description
protected void	abortMultiPartUploadInternal()
protected void	completeMultiPartUploadInternal()
protected void	createEmptyObject(String key)
protected com.amazonaws.services.s3.AmazonS3	getClient()
Optional<String>	getContentHash()
protected void	initMultiPartUploadInternal()
protected void	putObject(String key, File file, String md5)
protected void	uploadPartInternal(File file, int partNumber, boolean isLastPart, String md5)

Methods inherited from class alluxio.underfs.ObjectLowLevelOutputStream

abortMultiPartUpload, close, flush, getPartNumber, uploadPart, uploadPart, waitForAllPartsUpload, write, write, write

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

mClient

protected com.amazonaws.services.s3.AmazonS3 mClient

The Amazon S3 client to interact with S3.

mUploadId

protected volatile String mUploadId

The upload id of this multipart upload.

Constructor Detail

S3ALowLevelOutputStream

public S3ALowLevelOutputStream(String bucketName,
                               String key,
                               com.amazonaws.services.s3.AmazonS3 s3Client,
                               com.google.common.util.concurrent.ListeningExecutorService executor,
                               AlluxioConfiguration ufsConf)

Constructs a new stream for writing a file.

Parameters:

bucketName - the name of the bucket

key - the key of the file

s3Client - the Amazon S3 client to upload the file with

executor - a thread pool executor

ufsConf - the object store under file system configuration

Method Detail

uploadPartInternal

protected void uploadPartInternal(File file,
                                  int partNumber,
                                  boolean isLastPart,
                                  @Nullable
                                  String md5)
                           throws IOException

Specified by:

uploadPartInternal in class ObjectLowLevelOutputStream

Throws:

IOException

initMultiPartUploadInternal

protected void initMultiPartUploadInternal()
                                    throws IOException

Specified by:

initMultiPartUploadInternal in class ObjectLowLevelOutputStream

Throws:

IOException

completeMultiPartUploadInternal

protected void completeMultiPartUploadInternal()
                                        throws IOException

Specified by:

completeMultiPartUploadInternal in class ObjectLowLevelOutputStream

Throws:

IOException

abortMultiPartUploadInternal

protected void abortMultiPartUploadInternal()
                                     throws IOException

Specified by:

abortMultiPartUploadInternal in class ObjectLowLevelOutputStream

Throws:

IOException

createEmptyObject

protected void createEmptyObject(String key)
                          throws IOException

Specified by:

createEmptyObject in class ObjectLowLevelOutputStream

Throws:

IOException

putObject

protected void putObject(String key,
                         File file,
                         @Nullable
                         String md5)
                  throws IOException

Specified by:

putObject in class ObjectLowLevelOutputStream

Throws:

IOException

getClient

protected com.amazonaws.services.s3.AmazonS3 getClient()

getContentHash

public Optional<String> getContentHash()

Returns:

the content hash of the file written to the UFS if available after the stream has been closed