alluxio.underfs.s3a

Class S3ALowLevelOutputStream

java.lang.Object

java.io.OutputStream

alluxio.underfs.s3a.S3ALowLevelOutputStream

All Implemented Interfaces:

Closeable, Flushable, AutoCloseable

@NotThreadSafe
public class S3ALowLevelOutputStream
extends OutputStream

[Experimental] A stream for writing a file into S3 using streaming upload. The data transfer is done using S3 low-level multipart upload. The multipart upload is initialized in the first write() and an upload id is given by AWS S3 to distinguish different multipart uploads. We upload data in partitions. When write(), the data will be persisted to a temporary file mFile on the local disk. When the data mPartitionOffset in this temporary file reaches the mPartitionSize, the file will be submitted to the upload executor mExecutor and we do not wait for uploads to finish. A new temp file will be created for the future write and the mPartitionOffset will be reset to zero. The process goes until all the data has been written to temp files. In flush(), we upload the buffered data if they are bigger than 5MB and wait for all uploads to finish. The temp files will be deleted after uploading successfully. In close(), we upload the last part of data (if exists), wait for all uploads to finish, and complete the multipart upload. close() will not be retried, but all the multipart upload related operations(init, upload, complete, and abort) will be retried. If an error occurs and we have no way to recover, we abort the multipart uploads. Some multipart uploads may not be completed/aborted in normal ways and need periodical cleanup by enabling the PropertyKey.UNDERFS_CLEANUP_ENABLED. When a leader master starts or a cleanup interval is reached, all the multipart uploads older than PropertyKey.UNDERFS_S3_INTERMEDIATE_UPLOAD_CLEAN_AGE will be cleaned.

Constructor Summary

Constructors

Constructor and Description
S3ALowLevelOutputStream(String bucketName, String key, com.amazonaws.services.s3.AmazonS3 s3Client, com.google.common.util.concurrent.ListeningExecutorService executor, long streamingUploadPartitionSize, List<String> tmpDirs, boolean sseEnabled) Constructs a new stream for writing a file.

Method Summary

Modifier and Type	Method and Description
void	close()
void	flush()
void	write(byte[] b)
void	write(byte[] b, int off, int len)
void	write(int b)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

S3ALowLevelOutputStream

public S3ALowLevelOutputStream(String bucketName,
                               String key,
                               com.amazonaws.services.s3.AmazonS3 s3Client,
                               com.google.common.util.concurrent.ListeningExecutorService executor,
                               long streamingUploadPartitionSize,
                               List<String> tmpDirs,
                               boolean sseEnabled)

Constructs a new stream for writing a file.

Parameters:

bucketName - the name of the bucket

key - the key of the file

s3Client - the Amazon S3 client to upload the file with

executor - a thread pool executor

streamingUploadPartitionSize - the size in bytes for partitions of streaming uploads

tmpDirs - a list of temporary directories

sseEnabled - whether or not server side encryption is enabled

Method Detail

write

public void write(int b)
           throws IOException

Specified by:

write in class OutputStream

Throws:

IOException

write

public void write(byte[] b)
           throws IOException

Overrides:

write in class OutputStream

Throws:

IOException

write

public void write(byte[] b,
                  int off,
                  int len)
           throws IOException

Overrides:

write in class OutputStream

Throws:

IOException

flush

public void flush()
           throws IOException

Specified by:

flush in interface Flushable

Overrides:

flush in class OutputStream

Throws:

IOException

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Overrides:

close in class OutputStream

Throws:

IOException