alluxio.proxy.s3

Class S3RestUtils

java.lang.Object

alluxio.proxy.s3.S3RestUtils

public final class S3RestUtils
extends Object

Utilities for handling S3 REST calls.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	S3RestUtils.RestCallable<T> An interface representing a callable.
static class	S3RestUtils.URIStatusNameComparator Comparator based on uri name， treat uri name as a Long number.

Field Summary

Fields

Modifier and Type	Field and Description
static String	ALLUXIO_USER_HEADER
static String	MULTIPART_UPLOADS_METADATA_DIR

Method Summary

Modifier and Type	Method and Description
static <T> javax.ws.rs.core.Response	call(String resource, S3RestUtils.RestCallable<T> callable) Calls the given S3RestUtils.RestCallable and handles any exceptions thrown.
static void	checkPathIsAlluxioDirectory(FileSystem fs, String bucketPath, S3AuditContext auditContext) Check if a path in alluxio is a directory.
static void	checkPathIsAlluxioDirectory(FileSystem fs, String bucketPath, S3AuditContext auditContext, com.google.common.cache.Cache<AlluxioURI,Boolean> bucketPathCache) Check if a path in alluxio is a directory.
static List<URIStatus>	checkStatusesForUploadId(FileSystem metaFs, FileSystem userFs, AlluxioURI multipartTempDirUri, String uploadId) Fetches and returns the corresponding URIStatus for both the multipart upload temp directory and the Alluxio S3 metadata file.
static FileSystem	createFileSystemForUser(String user, FileSystem fs)
static Optional<com.google.common.util.concurrent.RateLimiter>	createRateLimiter(long rate) Create a rate limiter for given rate.
static void	deleteExistObject(FileSystem fs, AlluxioURI objectURI) Delete an existing key.
static void	deleteExistObject(FileSystem fs, AlluxioURI objectURI, Boolean recursive) Delete an existing key.
static javax.ws.rs.core.MediaType	deserializeContentType(Map<String,byte[]> xAttr) Given xAttr, parses and returns the Content-Type header metadata as its corresponding MediaType, or otherwise defaults to MediaType.APPLICATION_OCTET_STREAM_TYPE.
static TaggingData	deserializeTaggingHeader(String taggingHeader, int maxHeaderMetadataSize) Given a URL-encoded Tagging header, parses and deserializes the Tagging metadata into a TaggingData object.
static TaggingData	deserializeTags(Map<String,byte[]> xAttr) Given xAttr, parses and deserializes the Tagging metadata into a TaggingData object.
static Map<String,String>	fromMultiValueToSingleValueMap(javax.ws.rs.core.MultivaluedMap<String,String> queryParameters, boolean lowerCase) Convert MultivaluedMap to a single value map.
static String	getEntityTag(URIStatus status) This helper method is used to get the ETag xAttr on an object.
static String	getMultipartMetaFilepathForUploadId(String uploadId)
static String	getMultipartTemporaryDirForObject(String bucketPath, String objectKey, String uploadId)
static WritePType	getS3WriteType()
static String	getUser(String authorization, javax.ws.rs.container.ContainerRequestContext requestContext) Get username from header info.
static String	getUser(String authorization, javax.servlet.http.HttpServletRequest request) Get username from header info from HttpServletRequest.
static String	getUserFromAuthorization(String authorization, AlluxioConfiguration conf) Gets the user from the authorization header string for AWS Signature Version 4.
static boolean	isAuthenticationEnabled(AlluxioConfiguration conf) Checks if authentication is enabled.
static String	parsePath(String bucketPath) Format bucket path.
static void	populateContentTypeInXAttr(Map<String,com.google.protobuf.ByteString> xattrMap, String contentTypeHeader) Populate xattr with content type info from header.
static void	populateTaggingInXAttr(Map<String,com.google.protobuf.ByteString> xattrMap, String taggingHeader, S3AuditContext auditContext, String objectPath) Populate xattr map with tagging info from tagging header.
static void	setEntityTag(FileSystem fs, AlluxioURI objectUri, String entityTag) This helper method is used to set the ETag xAttr on an object.
static S3Exception	toBucketS3Exception(Exception exception, String resource) Convert an exception to instance of S3Exception.
static S3Exception	toBucketS3Exception(Exception exception, String resource, S3AuditContext auditContext) Convert an exception to instance of S3Exception.
static S3Exception	toObjectS3Exception(Exception exception, String resource) Convert an exception to instance of S3Exception.
static S3Exception	toObjectS3Exception(Exception exception, String resource, S3AuditContext auditContext) Convert an exception to instance of S3Exception.
static String	toS3Date(long epoch)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

ALLUXIO_USER_HEADER

public static final String ALLUXIO_USER_HEADER

See Also:

Constant Field Values

MULTIPART_UPLOADS_METADATA_DIR

public static final String MULTIPART_UPLOADS_METADATA_DIR

See Also:

Constant Field Values

Method Detail

call

public static <T> javax.ws.rs.core.Response call(String resource,
                                                 S3RestUtils.RestCallable<T> callable)

Calls the given S3RestUtils.RestCallable and handles any exceptions thrown.

Type Parameters:

T - the return type of the callable

Parameters:

resource - the resource (bucket or object) to be operated on

callable - the callable to call

Returns:

the response object

getMultipartTemporaryDirForObject

public static String getMultipartTemporaryDirForObject(String bucketPath,
                                                       String objectKey,
                                                       String uploadId)

Parameters:

bucketPath - the bucket path like "/bucket", "/mount/point/bucket"

objectKey - the object key like "img/2017/9/1/s3.jpg"

uploadId - the upload ID

Returns:

the temporary directory used to hold parts of the object during multipart uploads

getMultipartMetaFilepathForUploadId

public static String getMultipartMetaFilepathForUploadId(String uploadId)

Parameters:

uploadId - the upload ID

Returns:

the Alluxio UFS filepath containing the metadata for this upload

toS3Date

public static String toS3Date(long epoch)

Parameters:

epoch - the milliseconds from the epoch

Returns:

the string representation of the epoch in the S3 date format

parsePath

public static String parsePath(String bucketPath)

Format bucket path.

Parameters:

bucketPath - bucket path

Returns:

bucket path after format

toBucketS3Exception

public static S3Exception toBucketS3Exception(Exception exception,
                                              String resource)

Convert an exception to instance of S3Exception.

Parameters:

exception - Exception thrown when process s3 object rest request

resource - complete bucket path

Returns:

instance of S3Exception

toBucketS3Exception

public static S3Exception toBucketS3Exception(Exception exception,
                                              String resource,
                                              @Nonnull
                                              S3AuditContext auditContext)

Convert an exception to instance of S3Exception.

Parameters:

exception - Exception thrown when process s3 object rest request

resource - complete bucket path

auditContext - the audit context for exception

Returns:

instance of S3Exception

toObjectS3Exception

public static S3Exception toObjectS3Exception(Exception exception,
                                              String resource)

Convert an exception to instance of S3Exception.

Parameters:

exception - Exception thrown when process s3 object rest request

resource - object complete path

Returns:

instance of S3Exception

toObjectS3Exception

public static S3Exception toObjectS3Exception(Exception exception,
                                              String resource,
                                              @Nonnull
                                              S3AuditContext auditContext)

Convert an exception to instance of S3Exception.

Parameters:

exception - Exception thrown when process s3 object rest request

resource - object complete path

auditContext - the audit context for exception

Returns:

instance of S3Exception

checkPathIsAlluxioDirectory

public static void checkPathIsAlluxioDirectory(FileSystem fs,
                                               String bucketPath,
                                               @Nullable
                                               S3AuditContext auditContext)
                                        throws S3Exception

Check if a path in alluxio is a directory.

Parameters:

fs - instance of FileSystem

bucketPath - bucket complete path

auditContext - the audit context for exception

Throws:

S3Exception

checkPathIsAlluxioDirectory

public static void checkPathIsAlluxioDirectory(FileSystem fs,
                                               String bucketPath,
                                               @Nullable
                                               S3AuditContext auditContext,
                                               com.google.common.cache.Cache<AlluxioURI,Boolean> bucketPathCache)
                                        throws S3Exception

Check if a path in alluxio is a directory.

Parameters:

fs - instance of FileSystem

bucketPath - bucket complete path

auditContext - the audit context for exception

bucketPathCache - cache the bucket path for a certain time period

Throws:

S3Exception

checkStatusesForUploadId

public static List<URIStatus> checkStatusesForUploadId(FileSystem metaFs,
                                                       FileSystem userFs,
                                                       AlluxioURI multipartTempDirUri,
                                                       String uploadId)
                                                throws AlluxioException,
                                                       IOException

Fetches and returns the corresponding URIStatus for both the multipart upload temp directory and the Alluxio S3 metadata file.

Parameters:

metaFs - instance of FileSystem - used for metadata operations

userFs - instance of FileSystem - under the scope of a user agent

multipartTempDirUri - multipart upload tmp directory URI

uploadId - multipart upload Id

Returns:

a list of file statuses: - first, the multipart upload temp directory - second, the metadata file

Throws:

AlluxioException

IOException

deleteExistObject

public static void deleteExistObject(FileSystem fs,
                                     AlluxioURI objectURI)
                              throws IOException,
                                     AlluxioException

Delete an existing key.

Parameters:

fs - instance of FileSystem

objectURI - the key uri

Throws:

IOException

AlluxioException

deleteExistObject

public static void deleteExistObject(FileSystem fs,
                                     AlluxioURI objectURI,
                                     Boolean recursive)
                              throws IOException,
                                     AlluxioException

Delete an existing key.

Parameters:

fs - instance of FileSystem

objectURI - the key uri

recursive - if delete option is recursive

Throws:

IOException

AlluxioException

getS3WriteType

public static WritePType getS3WriteType()

Returns:

s3 WritePType

isAuthenticationEnabled

public static boolean isAuthenticationEnabled(AlluxioConfiguration conf)

Checks if authentication is enabled.

Parameters:

conf - Alluxio configuration

Returns:

true if authentication is enabled, false otherwise

createFileSystemForUser

public static FileSystem createFileSystemForUser(String user,
                                                 FileSystem fs)

Parameters:

user - the Subject name of the filesystem user

fs - the source FileSystem to base off of

Returns:

A FileSystem with the subject set to the provided user

fromMultiValueToSingleValueMap

public static Map<String,String> fromMultiValueToSingleValueMap(javax.ws.rs.core.MultivaluedMap<String,String> queryParameters,
                                                                boolean lowerCase)

Convert MultivaluedMap to a single value map.

Parameters:

queryParameters - MultivaluedMap

lowerCase - whether to use lower case

Returns:

a single value map

deserializeContentType

public static javax.ws.rs.core.MediaType deserializeContentType(Map<String,byte[]> xAttr)

Given xAttr, parses and returns the Content-Type header metadata as its corresponding MediaType, or otherwise defaults to MediaType.APPLICATION_OCTET_STREAM_TYPE.

Parameters:

xAttr - the Inode's xAttrs

Returns:

the MediaType corresponding to the Content-Type header

deserializeTags

public static TaggingData deserializeTags(Map<String,byte[]> xAttr)
                                   throws IOException

Given xAttr, parses and deserializes the Tagging metadata into a TaggingData object. Returns null if no data exists.

Parameters:

xAttr - the Inode's xAttrs

Returns:

the deserialized TaggingData object

Throws:

IOException

deserializeTaggingHeader

public static TaggingData deserializeTaggingHeader(String taggingHeader,
                                                   int maxHeaderMetadataSize)
                                            throws S3Exception

Given a URL-encoded Tagging header, parses and deserializes the Tagging metadata into a TaggingData object. Returns null on empty strings.

Parameters:

taggingHeader - the URL-encoded Tagging header

maxHeaderMetadataSize - Header user-metadata size limit validation (default max: 2 KB) - https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html

Returns:

the deserialized TaggingData object

Throws:

S3Exception

setEntityTag

public static void setEntityTag(FileSystem fs,
                                AlluxioURI objectUri,
                                String entityTag)
                         throws IOException,
                                AlluxioException

This helper method is used to set the ETag xAttr on an object.

Parameters:

fs - The FileSystem used to make the gRPC request

objectUri - The AlluxioURI for the object to update

entityTag - The entity tag of the object (MD5 checksum of the object contents)

Throws:

IOException

AlluxioException

getEntityTag

public static String getEntityTag(URIStatus status)

This helper method is used to get the ETag xAttr on an object.

Parameters:

status - The URIStatus of the object

Returns:

the entityTag String, or null if it does not exist

getUser

public static String getUser(String authorization,
                             javax.ws.rs.container.ContainerRequestContext requestContext)
                      throws S3Exception

Get username from header info.

Parameters:

authorization - authorization info

requestContext - request context

Returns:

username

Throws:

S3Exception

getUser

public static String getUser(String authorization,
                             javax.servlet.http.HttpServletRequest request)
                      throws S3Exception

Get username from header info from HttpServletRequest.

Parameters:

authorization -

request -

Returns:

user name

Throws:

S3Exception

getUserFromAuthorization

public static String getUserFromAuthorization(String authorization,
                                              AlluxioConfiguration conf)
                                       throws S3Exception

Gets the user from the authorization header string for AWS Signature Version 4.

Parameters:

authorization - the authorization header string

conf - the AlluxioConfiguration Alluxio conf

Returns:

the user

Throws:

S3Exception

populateContentTypeInXAttr

public static void populateContentTypeInXAttr(Map<String,com.google.protobuf.ByteString> xattrMap,
                                              String contentTypeHeader)

Populate xattr with content type info from header.

Parameters:

xattrMap -

contentTypeHeader -

populateTaggingInXAttr

public static void populateTaggingInXAttr(Map<String,com.google.protobuf.ByteString> xattrMap,
                                          String taggingHeader,
                                          S3AuditContext auditContext,
                                          String objectPath)
                                   throws S3Exception

Populate xattr map with tagging info from tagging header.

Parameters:

xattrMap -

taggingHeader -

auditContext -

objectPath -

Throws:

S3Exception

createRateLimiter

public static Optional<com.google.common.util.concurrent.RateLimiter> createRateLimiter(long rate)

Create a rate limiter for given rate.

Parameters:

rate - bytes per second

Returns:

empty if rate <= 0