@NotThreadSafe public class S3ALowLevelOutputStream extends OutputStream
mFile
on the local disk. When the data mPartitionOffset
in this temporary file reaches the mPartitionSize
, the file will be submitted
to the upload executor mExecutor
and we do not wait for uploads to finish.
A new temp file will be created for the future write and the mPartitionOffset
will be reset to zero. The process goes until all the data has been written to temp files.
In flush(), we upload the buffered data if they are bigger than 5MB
and wait for all uploads to finish. The temp files will be deleted after uploading successfully.
In close(), we upload the last part of data (if exists), wait for all uploads to finish,
and complete the multipart upload.
close() will not be retried, but all the multipart upload
related operations(init, upload, complete, and abort) will be retried.
If an error occurs and we have no way to recover, we abort the multipart uploads.
Some multipart uploads may not be completed/aborted in normal ways and need periodical cleanup
by enabling the PropertyKey.UNDERFS_CLEANUP_ENABLED
.
When a leader master starts or a cleanup interval is reached, all the multipart uploads
older than PropertyKey.UNDERFS_S3_INTERMEDIATE_UPLOAD_CLEAN_AGE
will be cleaned.Constructor and Description |
---|
S3ALowLevelOutputStream(String bucketName,
String key,
com.amazonaws.services.s3.AmazonS3 s3Client,
com.google.common.util.concurrent.ListeningExecutorService executor,
long streamingUploadPartitionSize,
List<String> tmpDirs,
boolean sseEnabled)
Constructs a new stream for writing a file.
|
Modifier and Type | Method and Description |
---|---|
void |
close() |
void |
flush() |
void |
write(byte[] b) |
void |
write(byte[] b,
int off,
int len) |
void |
write(int b) |
public S3ALowLevelOutputStream(String bucketName, String key, com.amazonaws.services.s3.AmazonS3 s3Client, com.google.common.util.concurrent.ListeningExecutorService executor, long streamingUploadPartitionSize, List<String> tmpDirs, boolean sseEnabled)
bucketName
- the name of the bucketkey
- the key of the files3Client
- the Amazon S3 client to upload the file withexecutor
- a thread pool executorstreamingUploadPartitionSize
- the size in bytes for partitions of streaming uploadstmpDirs
- a list of temporary directoriessseEnabled
- whether or not server side encryption is enabledpublic void write(int b) throws IOException
write
in class OutputStream
IOException
public void write(byte[] b) throws IOException
write
in class OutputStream
IOException
public void write(byte[] b, int off, int len) throws IOException
write
in class OutputStream
IOException
public void flush() throws IOException
flush
in interface Flushable
flush
in class OutputStream
IOException
public void close() throws IOException
close
in interface Closeable
close
in interface AutoCloseable
close
in class OutputStream
IOException
Copyright © 2023. All Rights Reserved.