alluxio.underfs.s3a

Class S3AOutputStream

java.lang.Object

java.io.OutputStream

alluxio.underfs.s3a.S3AOutputStream

All Implemented Interfaces:

ContentHashable, Closeable, Flushable, AutoCloseable

@NotThreadSafe
public class S3AOutputStream
extends OutputStream
implements ContentHashable

A stream for writing a file into S3. The data will be persisted to a temporary directory on the local disk and copied as a complete file when the close() method is called. The data transfer is done using a TransferManager which manages the upload threads and handles multipart upload.

Field Summary

Fields

Modifier and Type	Field and Description
protected String	mKey Key of the file when it is uploaded to S3.
protected com.amazonaws.services.s3.transfer.TransferManager	mManager A TransferManager to upload the file to S3 using Multipart Uploads.

Constructor Summary

Constructors

Constructor and Description
S3AOutputStream(String bucketName, String key, com.amazonaws.services.s3.transfer.TransferManager manager, List<String> tmpDirs, boolean sseEnabled) Constructs a new stream for writing a file.

Method Summary

Modifier and Type	Method and Description
void	close()
void	flush()
Optional<String>	getContentHash()
protected com.amazonaws.services.s3.transfer.TransferManager	getTransferManager()
protected String	getUploadPath()
void	write(byte[] b)
void	write(byte[] b, int off, int len)
void	write(int b)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

mKey

protected final String mKey

Key of the file when it is uploaded to S3.

mManager

protected com.amazonaws.services.s3.transfer.TransferManager mManager

A TransferManager to upload the file to S3 using Multipart Uploads. Multipart Uploads involves uploading an object's data in parts instead of all at once, which can work around S3's limit of 5GB on a single Object PUT operation. It is recommended (http://docs.aws.amazon.com/AmazonS3/latest/dev/UploadingObjects.html) to upload file larger than 100MB using Multipart Uploads.

Constructor Detail

S3AOutputStream

public S3AOutputStream(String bucketName,
                       String key,
                       com.amazonaws.services.s3.transfer.TransferManager manager,
                       List<String> tmpDirs,
                       boolean sseEnabled)
                throws IOException

Constructs a new stream for writing a file.

Parameters:

bucketName - the name of the bucket

key - the key of the file

manager - the transfer manager to upload the file with

tmpDirs - a list of temporary directories

sseEnabled - whether or not server side encryption is enabled

Throws:

IOException

Method Detail

write

public void write(int b)
           throws IOException

Specified by:

write in class OutputStream

Throws:

IOException

write

public void write(byte[] b)
           throws IOException

Overrides:

write in class OutputStream

Throws:

IOException

write

public void write(byte[] b,
                  int off,
                  int len)
           throws IOException

Overrides:

write in class OutputStream

Throws:

IOException

flush

public void flush()
           throws IOException

Specified by:

flush in interface Flushable

Overrides:

flush in class OutputStream

Throws:

IOException

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Overrides:

close in class OutputStream

Throws:

IOException

getUploadPath

protected String getUploadPath()

Returns:

the path in S3 to upload the file to

getTransferManager

protected com.amazonaws.services.s3.transfer.TransferManager getTransferManager()

Returns:

return the transfer manager

getContentHash

public Optional<String> getContentHash()

Specified by:

getContentHash in interface ContentHashable

Returns:

the content hash of the file written to the UFS if available after the stream has been closed