alluxio.proxy.s3

Class S3RestUtils

java.lang.Object

alluxio.proxy.s3.S3RestUtils

public final class S3RestUtils
extends Object

Utilities for handling S3 REST calls.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	S3RestUtils.RestCallable<T> An interface representing a callable.
static class	S3RestUtils.URIStatusNameComparator Comparator based on uri name， treat uri name as a Long number.

Field Summary

Fields

Modifier and Type	Field and Description
static String	MULTIPART_UPLOADS_METADATA_DIR

Method Summary

Modifier and Type	Method and Description
static <T> javax.ws.rs.core.Response	call(String resource, S3RestUtils.RestCallable<T> callable) Calls the given S3RestUtils.RestCallable and handles any exceptions thrown.
static void	checkPathIsAlluxioDirectory(FileSystem fs, String bucketPath) Check if a path in alluxio is a directory.
static List<URIStatus>	checkStatusesForUploadId(FileSystem fs, AlluxioURI multipartTempDirUri, String uploadId) Fetches and returns the corresponding URIStatus for both the multipart upload temp directory and the Alluxio S3 metadata file.
static void	deleteExistObject(FileSystem fs, AlluxioURI objectURI) Delete an existing key.
static void	deleteExistObject(FileSystem fs, AlluxioURI objectURI, Boolean recursive) Delete an existing key.
static javax.ws.rs.core.MediaType	deserializeContentType(Map<String,byte[]> xAttr) Given xAttr, parses and returns the Content-Type header metadata as its corresponding MediaType, or otherwise defaults to MediaType.APPLICATION_OCTET_STREAM_TYPE.
static TaggingData	deserializeTaggingHeader(String taggingHeader, int maxHeaderMetadataSize) Given a URL-encoded Tagging header, parses and deserializes the Tagging metadata into a TaggingData object.
static TaggingData	deserializeTags(Map<String,byte[]> xAttr) Given xAttr, parses and deserializes the Tagging metadata into a TaggingData object.
static Map<String,String>	fromMultiValueToSingleValueMap(javax.ws.rs.core.MultivaluedMap<String,String> queryParameters, boolean lowerCase) Convert MultivaluedMap to a single value map.
static String	getEntityTag(URIStatus status) This helper method is used to get the ETag xAttr on an object.
static String	getMultipartMetaFilepathForUploadId(String uploadId)
static String	getMultipartTemporaryDirForObject(String bucketPath, String objectKey, String uploadId)
static WritePType	getS3WriteType()
static boolean	isAuthenticationEnabled(AlluxioConfiguration conf) Checks if authentication is enabled.
static String	parsePath(String bucketPath) Format bucket path.
static void	setEntityTag(FileSystem fs, AlluxioURI objectUri, String entityTag) This helper method is used to set the ETag xAttr on an object.
static S3Exception	toBucketS3Exception(Exception exception, String resource) Convert an exception to instance of S3Exception.
static S3Exception	toObjectS3Exception(Exception exception, String resource) Convert an exception to instance of S3Exception.
static String	toS3Date(long epoch)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MULTIPART_UPLOADS_METADATA_DIR

public static final String MULTIPART_UPLOADS_METADATA_DIR

See Also:

Constant Field Values

Method Detail

call

public static <T> javax.ws.rs.core.Response call(String resource,
                                                 S3RestUtils.RestCallable<T> callable)

Calls the given S3RestUtils.RestCallable and handles any exceptions thrown.

Type Parameters:

T - the return type of the callable

Parameters:

resource - the resource (bucket or object) to be operated on

callable - the callable to call

Returns:

the response object

getMultipartTemporaryDirForObject

public static String getMultipartTemporaryDirForObject(String bucketPath,
                                                       String objectKey,
                                                       String uploadId)

Parameters:

bucketPath - the bucket path like "/bucket", "/mount/point/bucket"

objectKey - the object key like "img/2017/9/1/s3.jpg"

uploadId - the upload ID

Returns:

the temporary directory used to hold parts of the object during multipart uploads

getMultipartMetaFilepathForUploadId

public static String getMultipartMetaFilepathForUploadId(String uploadId)

Parameters:

uploadId - the upload ID

Returns:

the Alluxio UFS filepath containing the metadata for this upload

toS3Date

public static String toS3Date(long epoch)

Parameters:

epoch - the milliseconds from the epoch

Returns:

the string representation of the epoch in the S3 date format

parsePath

public static String parsePath(String bucketPath)

Format bucket path.

Parameters:

bucketPath - bucket path

Returns:

bucket path after format

toBucketS3Exception

public static S3Exception toBucketS3Exception(Exception exception,
                                              String resource)

Convert an exception to instance of S3Exception.

Parameters:

exception - Exception thrown when process s3 object rest request

resource - complete bucket path

Returns:

instance of S3Exception

toObjectS3Exception

public static S3Exception toObjectS3Exception(Exception exception,
                                              String resource)

Convert an exception to instance of S3Exception.

Parameters:

exception - Exception thrown when process s3 object rest request

resource - object complete path

Returns:

instance of S3Exception

checkPathIsAlluxioDirectory

public static void checkPathIsAlluxioDirectory(FileSystem fs,
                                               String bucketPath)
                                        throws S3Exception

Check if a path in alluxio is a directory.

Parameters:

fs - instance of FileSystem

bucketPath - bucket complete path

Throws:

S3Exception

checkStatusesForUploadId

public static List<URIStatus> checkStatusesForUploadId(FileSystem fs,
                                                       AlluxioURI multipartTempDirUri,
                                                       String uploadId)
                                                throws AlluxioException,
                                                       IOException

Fetches and returns the corresponding URIStatus for both the multipart upload temp directory and the Alluxio S3 metadata file.

Parameters:

fs - instance of FileSystem

multipartTempDirUri - multipart upload tmp directory URI

uploadId - multipart upload Id

Returns:

a list of file statuses: - first, the multipart upload temp directory - second, the metadata file

Throws:

AlluxioException

IOException

deleteExistObject

public static void deleteExistObject(FileSystem fs,
                                     AlluxioURI objectURI)
                              throws IOException,
                                     AlluxioException

Delete an existing key.

Parameters:

fs - instance of FileSystem

objectURI - the key uri

Throws:

IOException

AlluxioException

deleteExistObject

public static void deleteExistObject(FileSystem fs,
                                     AlluxioURI objectURI,
                                     Boolean recursive)
                              throws IOException,
                                     AlluxioException

Delete an existing key.

Parameters:

fs - instance of FileSystem

objectURI - the key uri

recursive - if delete option is recursive

Throws:

IOException

AlluxioException

getS3WriteType

public static WritePType getS3WriteType()

Returns:

s3 WritePType

isAuthenticationEnabled

public static boolean isAuthenticationEnabled(AlluxioConfiguration conf)

Checks if authentication is enabled.

Parameters:

conf - Alluxio configuration

Returns:

true if authentication is enabled, false otherwise

fromMultiValueToSingleValueMap

public static Map<String,String> fromMultiValueToSingleValueMap(javax.ws.rs.core.MultivaluedMap<String,String> queryParameters,
                                                                boolean lowerCase)

Convert MultivaluedMap to a single value map.

Parameters:

queryParameters - MultivaluedMap

lowerCase - whether to use lower case

Returns:

a single value map

deserializeContentType

public static javax.ws.rs.core.MediaType deserializeContentType(Map<String,byte[]> xAttr)

Given xAttr, parses and returns the Content-Type header metadata as its corresponding MediaType, or otherwise defaults to MediaType.APPLICATION_OCTET_STREAM_TYPE.

Parameters:

xAttr - the Inode's xAttrs

Returns:

the MediaType corresponding to the Content-Type header

deserializeTags

public static TaggingData deserializeTags(Map<String,byte[]> xAttr)
                                   throws IOException

Given xAttr, parses and deserializes the Tagging metadata into a TaggingData object. Returns null if no data exists.

Parameters:

xAttr - the Inode's xAttrs

Returns:

the deserialized TaggingData object

Throws:

IOException

deserializeTaggingHeader

public static TaggingData deserializeTaggingHeader(String taggingHeader,
                                                   int maxHeaderMetadataSize)
                                            throws S3Exception

Given a URL-encoded Tagging header, parses and deserializes the Tagging metadata into a TaggingData object. Returns null on empty strings.

Parameters:

taggingHeader - the URL-encoded Tagging header

maxHeaderMetadataSize - Header user-metadata size limit validation (default max: 2 KB) - https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html

Returns:

the deserialized TaggingData object

Throws:

S3Exception

setEntityTag

public static void setEntityTag(FileSystem fs,
                                AlluxioURI objectUri,
                                String entityTag)
                         throws IOException,
                                AlluxioException

This helper method is used to set the ETag xAttr on an object.

Parameters:

fs - The FileSystem used to make the gRPC request

objectUri - The AlluxioURI for the object to update

entityTag - The entity tag of the object (MD5 checksum of the object contents)

Throws:

IOException

AlluxioException

getEntityTag

public static String getEntityTag(URIStatus status)

This helper method is used to get the ETag xAttr on an object.

Parameters:

status - The URIStatus of the object

Returns:

the entityTag String, or null if it does not exist