alluxio.underfs

Class ObjectMultipartUploadOutputStream

java.lang.Object

java.io.OutputStream

alluxio.underfs.ObjectMultipartUploadOutputStream

All Implemented Interfaces:

ContentHashable, Closeable, Flushable, AutoCloseable

Direct Known Subclasses:

COSMultipartUploadOutputStream, OBSMultipartUploadOutputStream, OSSMultipartUploadOutputStream, S3AMultipartUploadOutputStream

@NotThreadSafe
public abstract class ObjectMultipartUploadOutputStream
extends OutputStream
implements ContentHashable

Upload File to object storage in multiple parts.

The file is partitioned into multiple parts, each part is uploaded in a separate thread. The number of threads is determined by the configuration. The minimum part size is 5MB (s3) or 100KB (oss, obs) or 1MB(cos), except the last part. We choose 5MB to be the minimum part size for all object storage systems. The maximum part size is 5GB (s3, oss, cos, obs). The partition size is determined by the configuration.

In flush(), we wait for all uploads to finish.

In close() we complete the multipart upload.

close() will not be retried, but all the multipart upload related operations(init, upload, complete, and abort) will be retried.

If an error occurs, and we have no way to recover, we abort the multipart uploads. Some multipart uploads may not be completed/aborted in normal ways and need periodical cleanup by enabling the PropertyKey.UNDERFS_CLEANUP_ENABLED. When a leader master starts or a cleanup interval is reached, all the multipart uploads older than clean age will be cleaned.

Field Summary

Fields

Modifier and Type	Field and Description
protected static org.slf4j.Logger	LOG
protected static long	MAXIMUM_PART_SIZE The maximum size of a single part is 5GB.
protected String	mBucketName Bucket name of the object storage bucket.
protected boolean	mClosed Flag to indicate this stream has been closed, to ensure close is only done once.
protected static long	MINIMUM_PART_SIZE Only parts bigger than 5MB could be uploaded through multipart upload, except the last part.
protected String	mKey Key of the file when it is uploaded to object storage.
protected long	mPartitionOffset When the offset reaches the partition size, we upload mStream.
protected long	mPartitionSize The maximum allowed size of a partition.
protected java.util.function.Supplier<RetryPolicy>	mRetryPolicy The retry policy of this multipart upload.
protected byte[]	mSingleCharWrite Pre-allocated byte buffer for writing single characters.

Constructor Summary

Constructors

Constructor and Description
ObjectMultipartUploadOutputStream(String bucketName, String key, com.google.common.util.concurrent.ListeningExecutorService executor, long multipartUploadPartitionSize, AlluxioConfiguration ufsConf) Constructs a new stream for writing a file.

Method Summary

Modifier and Type	Method and Description
protected void	abortMultiPartUpload()
protected abstract void	abortMultipartUploadInternal()
void	close()
protected abstract void	completeMultipartUploadInternal()
protected abstract void	createEmptyObject(String key)
void	flush()
protected abstract void	initMultipartUploadInternal()
protected abstract void	putObject(String key, byte[] buf, long length)
protected void	uploadPart() Uploads part async.
protected void	uploadPart(byte[] buf, int partNumber, boolean isLastPart, long length)
protected abstract void	uploadPartInternal(byte[] buf, int partNumber, boolean isLastPart, long length)
protected void	waitForAllPartsUpload()
void	write(byte[] b)
void	write(byte[] b, int off, int len)
void	write(int b)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface alluxio.underfs.ContentHashable

getContentHash

Field Detail

LOG

protected static final org.slf4j.Logger LOG

MINIMUM_PART_SIZE

protected static final long MINIMUM_PART_SIZE

Only parts bigger than 5MB could be uploaded through multipart upload, except the last part.

See Also:

Constant Field Values

MAXIMUM_PART_SIZE

protected static final long MAXIMUM_PART_SIZE

The maximum size of a single part is 5GB.

See Also:

Constant Field Values

mBucketName

protected final String mBucketName

Bucket name of the object storage bucket.

mKey

protected final String mKey

Key of the file when it is uploaded to object storage.

mRetryPolicy

protected final java.util.function.Supplier<RetryPolicy> mRetryPolicy

The retry policy of this multipart upload.

mSingleCharWrite

protected final byte[] mSingleCharWrite

Pre-allocated byte buffer for writing single characters.

mPartitionSize

protected final long mPartitionSize

The maximum allowed size of a partition.

mClosed

protected boolean mClosed

Flag to indicate this stream has been closed, to ensure close is only done once.

mPartitionOffset

protected long mPartitionOffset

When the offset reaches the partition size, we upload mStream.

Constructor Detail

ObjectMultipartUploadOutputStream

public ObjectMultipartUploadOutputStream(String bucketName,
                                         String key,
                                         com.google.common.util.concurrent.ListeningExecutorService executor,
                                         long multipartUploadPartitionSize,
                                         AlluxioConfiguration ufsConf)

Constructs a new stream for writing a file.

Parameters:

bucketName - the name of the bucket

key - the key of the file

executor - executor

multipartUploadPartitionSize - the size in bytes for partitions of multipart uploads

ufsConf - the object store under file system configuration

Method Detail

write

public void write(int b)
           throws IOException

Specified by:

write in class OutputStream

Throws:

IOException

write

public void write(byte[] b)
           throws IOException

Overrides:

write in class OutputStream

Throws:

IOException

write

public void write(byte[] b,
                  int off,
                  int len)
           throws IOException

Overrides:

write in class OutputStream

Throws:

IOException

flush

public void flush()
           throws IOException

Specified by:

flush in interface Flushable

Overrides:

flush in class OutputStream

Throws:

IOException

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Overrides:

close in class OutputStream

Throws:

IOException

uploadPart

protected void uploadPart()
                   throws IOException

Uploads part async.

Throws:

IOException

uploadPart

protected void uploadPart(byte[] buf,
                          int partNumber,
                          boolean isLastPart,
                          long length)
                   throws IOException

Throws:

IOException

abortMultiPartUpload

protected void abortMultiPartUpload()
                             throws IOException

Throws:

IOException

waitForAllPartsUpload

protected void waitForAllPartsUpload()
                              throws IOException

Throws:

IOException

uploadPartInternal

protected abstract void uploadPartInternal(byte[] buf,
                                           int partNumber,
                                           boolean isLastPart,
                                           long length)
                                    throws IOException

Parameters:

buf - the byte buf

partNumber - the part number

isLastPart - whether this is the last part

length - the length of the part to be uploaded

Throws:

IOException

initMultipartUploadInternal

protected abstract void initMultipartUploadInternal()
                                             throws IOException

Throws:

IOException

completeMultipartUploadInternal

protected abstract void completeMultipartUploadInternal()
                                                 throws IOException

Throws:

IOException

abortMultipartUploadInternal

protected abstract void abortMultipartUploadInternal()
                                              throws IOException

Throws:

IOException

createEmptyObject

protected abstract void createEmptyObject(String key)
                                   throws IOException

Parameters:

key - the key

Throws:

IOException

putObject

protected abstract void putObject(String key,
                                  byte[] buf,
                                  long length)
                           throws IOException

Parameters:

key - the key

buf - the byte buf

length - the length of the file to be uploaded

Throws:

IOException