alluxio.underfs.s3a

Class S3AMultipartUploadOutputStream

java.lang.Object

java.io.OutputStream

alluxio.underfs.ObjectMultipartUploadOutputStream

alluxio.underfs.s3a.S3AMultipartUploadOutputStream

All Implemented Interfaces:

ContentHashable, Closeable, Flushable, AutoCloseable

@NotThreadSafe
public class S3AMultipartUploadOutputStream
extends ObjectMultipartUploadOutputStream

Object storage multipart upload for aws s3.

Field Summary

Fields

Modifier and Type	Field and Description
protected String	mUploadId The upload id of this multipart upload.

Fields inherited from class alluxio.underfs.ObjectMultipartUploadOutputStream

MAXIMUM_PART_SIZE, mBucketName, mClosed, MINIMUM_PART_SIZE, mKey, mPartitionOffset, mPartitionSize, mRetryPolicy, mSingleCharWrite

Constructor Summary

Constructors

Constructor and Description
S3AMultipartUploadOutputStream(String bucketName, String key, com.amazonaws.services.s3.AmazonS3 s3Client, com.google.common.util.concurrent.ListeningExecutorService executor, AlluxioConfiguration ufsConf) Constructs a new stream for writing a file.

Method Summary

Modifier and Type	Method and Description
protected void	abortMultipartUploadInternal()
protected void	completeMultipartUploadInternal()
protected void	createEmptyObject(String key)
protected com.amazonaws.services.s3.AmazonS3	getClient()
Optional<String>	getContentHash()
protected void	initMultipartUploadInternal()
protected void	putObject(String key, byte[] buf, long length)
protected void	uploadPartInternal(byte[] buf, int partNumber, boolean isLastPart, long length)

Methods inherited from class alluxio.underfs.ObjectMultipartUploadOutputStream

abortMultiPartUpload, close, flush, uploadPart, uploadPart, waitForAllPartsUpload, write, write, write

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

mUploadId

protected volatile String mUploadId

The upload id of this multipart upload.

Constructor Detail

S3AMultipartUploadOutputStream

public S3AMultipartUploadOutputStream(String bucketName,
                                      String key,
                                      com.amazonaws.services.s3.AmazonS3 s3Client,
                                      com.google.common.util.concurrent.ListeningExecutorService executor,
                                      AlluxioConfiguration ufsConf)

Constructs a new stream for writing a file.

Parameters:

bucketName - the name of the bucket

key - the key of the file

s3Client - the Amazon S3 client to upload the file with

executor - a thread pool executor

ufsConf - the object store under file system configuration

Method Detail

uploadPartInternal

protected void uploadPartInternal(byte[] buf,
                                  int partNumber,
                                  boolean isLastPart,
                                  long length)
                           throws IOException

Specified by:

uploadPartInternal in class ObjectMultipartUploadOutputStream

Parameters:

buf - the byte buf

partNumber - the part number

isLastPart - whether this is the last part

length - the length of the part to be uploaded

Throws:

IOException

initMultipartUploadInternal

protected void initMultipartUploadInternal()
                                    throws IOException

Specified by:

initMultipartUploadInternal in class ObjectMultipartUploadOutputStream

Throws:

IOException

completeMultipartUploadInternal

protected void completeMultipartUploadInternal()
                                        throws IOException

Specified by:

completeMultipartUploadInternal in class ObjectMultipartUploadOutputStream

Throws:

IOException

abortMultipartUploadInternal

protected void abortMultipartUploadInternal()
                                     throws IOException

Specified by:

abortMultipartUploadInternal in class ObjectMultipartUploadOutputStream

Throws:

IOException

createEmptyObject

protected void createEmptyObject(String key)
                          throws IOException

Specified by:

createEmptyObject in class ObjectMultipartUploadOutputStream

Parameters:

key - the key

Throws:

IOException

putObject

protected void putObject(String key,
                         byte[] buf,
                         long length)
                  throws IOException

Specified by:

putObject in class ObjectMultipartUploadOutputStream

Parameters:

key - the key

buf - the byte buf

length - the length of the file to be uploaded

Throws:

IOException

getClient

protected com.amazonaws.services.s3.AmazonS3 getClient()

getContentHash

public Optional<String> getContentHash()

Returns:

the content hash of the file written to the UFS if available after the stream has been closed