alluxio.util

Class CommonUtils

java.lang.Object

alluxio.util.CommonUtils

@ThreadSafe
public final class CommonUtils
extends Object

Common utilities shared by all components in Alluxio.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	CommonUtils.ProcessType Alluxio process types.

Field Summary

Fields

Modifier and Type	Field and Description
static AtomicReference<CommonUtils.ProcessType>	PROCESS_TYPE Represents the type of Alluxio process running in this JVM.

Method Summary

Modifier and Type	Method and Description
static <T> String	argsToString(String separator, T... args) Converts varargs of objects to a string.
static IOException	castToIOException(Throwable e) Casts a Throwable to an IOException.
static RuntimeException	closeAndRethrow(com.google.common.io.Closer closer, Throwable t) Closes the Closer and re-throws the Throwable.
static Map<String,byte[]>	convertFromByteString(Map<String,com.google.protobuf.ByteString> input)
static String	convertMsToClockTime(long millis) Converts milliseconds to clock time.
static String	convertMsToDate(long millis, String dateFormatPattern) Converts a millisecond number to a formatted date String.
static Map<String,com.google.protobuf.ByteString>	convertToByteString(Map<String,byte[]> input)
static <T> T	createNewClassInstance(Class<T> cls, Class<?>[] ctorClassArgs, Object[] ctorArgs) Creates new instance of a class by calling a constructor that receives ctorClassArgs arguments.
static Thread	createProgressThread(long intervalMs, PrintStream stream) Creates a thread which will write "." to the given print stream at the given interval.
static Thread	createProgressThread(PrintStream stream) Convenience method for calling createProgressThread(long, PrintStream) with an interval of 2 seconds.
static long	getCurrentMs()
static List<String>	getGroups(String userName, AlluxioConfiguration conf) Using CachedGroupMapping to get the group list of a user.
static String	getPrimaryGroupName(String userName, AlluxioConfiguration conf) Gets the primary group name of a user.
static Throwable	getRootCause(Throwable e) Gets the root cause of an exception.
static String	getTmpDir(List<String> tmpDirs)
static List<String>	getUnixGroups(String user) Gets the current user's group list from Unix by running the command 'groups' NOTE.
static String	getValueFromStaticMapping(String mapping, String key) Gets the value with a given key from a static key/value mapping in string format.
static String	getWorkerDataDirectory(String storageDir, AlluxioConfiguration conf)
static <T> void	invokeAll(List<Callable<T>> callables, long timeoutMs) Executes the given callables, waiting for them to complete (or time out).
static boolean	isLocalHost(WorkerNetAddress address, AlluxioConfiguration conf)
static <T> String	listToString(List<T> list) Converts a list of objects to a string.
static <T> java.util.function.Supplier<T>	memoize(java.util.function.Supplier<T> original) Memoize implementation for java.util.function.supplier.
static String	randomAlphaNumString(int length) Generates a random alphanumeric string of the given length.
static byte[]	randomBytes(int length) Generates a random byte array of the given length.
static <T> Iterator<T>	singleElementIterator(T element) Returns an iterator that iterates on a single element.
static void	sleepMs(org.slf4j.Logger logger, long timeMs) Sleeps for the given number of milliseconds, reporting interruptions using the given logger.
static void	sleepMs(long timeMs) Sleeps for the given number of milliseconds.
static String	stripLeadingAndTrailingQuotes(String str) Strips the leading and trailing quotes from the given string.
static String	stripPrefixIfPresent(String key, String prefix) Strips the prefix from the key if it is present.
static String	stripSuffixIfPresent(String key, String suffix) Strips the suffix if it exists.
static String[]	toStringArray(ArrayList<String> src) Parses ArrayList<String> into String[].
static void	unwrapResponse(Protocol.Response response) Unwraps a Protocol.Response.
static void	unwrapResponseFrom(Protocol.Response response, io.netty.channel.Channel channel) Unwraps a Protocol.Response associated with a channel.
static void	waitFor(String description, java.util.function.Supplier<Boolean> condition) Waits for a condition to be satisfied.
static void	waitFor(String description, java.util.function.Supplier<Boolean> condition, WaitForOptions options) Waits for a condition to be satisfied.
static <T> T	waitForResult(String description, java.util.function.Supplier<T> operation, WaitForOptions options) Waits for an operation to return a non-null value with a specified timeout.
static void	warmUpLoop() Common empty loop utility that serves the purpose of warming up the JVM before performance microbenchmarks.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PROCESS_TYPE

public static final AtomicReference<CommonUtils.ProcessType> PROCESS_TYPE

Represents the type of Alluxio process running in this JVM. NOTE: This will only be set by main methods of Alluxio processes. It will not be set properly for tests. Avoid using this field if at all possible.

Method Detail

createProgressThread

public static Thread createProgressThread(PrintStream stream)

Convenience method for calling createProgressThread(long, PrintStream) with an interval of 2 seconds.

Parameters:

stream - the print stream to write to

Returns:

the thread

createProgressThread

public static Thread createProgressThread(long intervalMs,
                                          PrintStream stream)

Creates a thread which will write "." to the given print stream at the given interval. The created thread is not started by this method. The created thread will be a daemon thread and will halt when interrupted.

Parameters:

intervalMs - the time interval in milliseconds between writes

stream - the print stream to write to

Returns:

the thread

getCurrentMs

public static long getCurrentMs()

Returns:

current time in milliseconds

getTmpDir

public static String getTmpDir(List<String> tmpDirs)

Parameters:

tmpDirs - the list of possible temporary directories to pick from

Returns:

a path to a temporary directory based on the user configuration

getWorkerDataDirectory

public static String getWorkerDataDirectory(String storageDir,
                                            AlluxioConfiguration conf)

Parameters:

storageDir - the root of a storage directory in tiered storage

conf - Alluxio's current configuration

Returns:

the worker data folder path after each storage directory, the final path will be like "/mnt/ramdisk/alluxioworker" for storage dir "/mnt/ramdisk" by appending {@link PropertyKey#WORKER_DATA_FOLDER).

listToString

public static <T> String listToString(List<T> list)

Converts a list of objects to a string.

Type Parameters:

T - type of the objects

Parameters:

list - list of objects

Returns:

space-separated concatenation of the string representation returned by Object#toString of the individual objects

argsToString

public static <T> String argsToString(String separator,
                                      T... args)

Converts varargs of objects to a string.

Type Parameters:

T - type of the objects

Parameters:

separator - separator string

args - variable arguments

Returns:

concatenation of the string representation returned by Object#toString of the individual objects

toStringArray

public static String[] toStringArray(ArrayList<String> src)

Parses ArrayList<String> into String[].

Parameters:

src - is the ArrayList of strings

Returns:

an array of strings

randomAlphaNumString

public static String randomAlphaNumString(int length)

Generates a random alphanumeric string of the given length.

Parameters:

length - the length

Returns:

a random string

randomBytes

public static byte[] randomBytes(int length)

Generates a random byte array of the given length.

Parameters:

length - the length

Returns:

a random byte array

sleepMs

public static void sleepMs(long timeMs)

Sleeps for the given number of milliseconds.

Parameters:

timeMs - sleep duration in milliseconds

sleepMs

public static void sleepMs(org.slf4j.Logger logger,
                           long timeMs)

Sleeps for the given number of milliseconds, reporting interruptions using the given logger. Unlike Thread.sleep(), this method responds to interrupts by setting the thread interrupt status. This means that callers must check the interrupt status if they need to handle interrupts.

Parameters:

logger - logger for reporting interruptions; no reporting is done if the logger is null

timeMs - sleep duration in milliseconds

warmUpLoop

public static void warmUpLoop()

Common empty loop utility that serves the purpose of warming up the JVM before performance microbenchmarks.

createNewClassInstance

public static <T> T createNewClassInstance(Class<T> cls,
                                           Class<?>[] ctorClassArgs,
                                           Object[] ctorArgs)

Creates new instance of a class by calling a constructor that receives ctorClassArgs arguments.

Type Parameters:

T - type of the object

Parameters:

cls - the class to create

ctorClassArgs - parameters type list of the constructor to initiate, if null default constructor will be called

ctorArgs - the arguments to pass the constructor

Returns:

new class object

Throws:

RuntimeException - if the class cannot be instantiated

getUnixGroups

public static List<String> getUnixGroups(String user)
                                  throws IOException

Gets the current user's group list from Unix by running the command 'groups' NOTE. For non-existing user it will return EMPTY list. This method may return duplicate groups.

Parameters:

user - user name

Returns:

the groups list that the user belongs to. The primary group is returned first

Throws:

IOException

waitFor

public static void waitFor(String description,
                           java.util.function.Supplier<Boolean> condition)
                    throws InterruptedException,
                           TimeoutException

Waits for a condition to be satisfied.

Parameters:

description - a description of what causes condition to evaluate to true

condition - the condition to wait on

Throws:

TimeoutException - if the function times out while waiting for the condition to be true

InterruptedException

waitFor

public static void waitFor(String description,
                           java.util.function.Supplier<Boolean> condition,
                           WaitForOptions options)
                    throws InterruptedException,
                           TimeoutException

Waits for a condition to be satisfied.

Parameters:

description - a description of what causes condition to evaluate to true

condition - the condition to wait on

options - the options to use

Throws:

TimeoutException - if the function times out while waiting for the condition to be true

InterruptedException

waitForResult

public static <T> T waitForResult(String description,
                                  java.util.function.Supplier<T> operation,
                                  WaitForOptions options)
                           throws InterruptedException,
                                  TimeoutException

Waits for an operation to return a non-null value with a specified timeout.

Type Parameters:

T - the type of the return value

Parameters:

description - the description of this operation

operation - the operation

options - the options to use

Returns:

the first non-null value generated by the operation

Throws:

TimeoutException - if the function times out while waiting to get a non-null value

InterruptedException

getPrimaryGroupName

public static String getPrimaryGroupName(String userName,
                                         AlluxioConfiguration conf)
                                  throws IOException

Gets the primary group name of a user.

Parameters:

userName - Alluxio user name

conf - Alluxio configuration

Returns:

primary group name

Throws:

IOException

getGroups

public static List<String> getGroups(String userName,
                                     AlluxioConfiguration conf)
                              throws IOException

Using CachedGroupMapping to get the group list of a user.

Parameters:

userName - Alluxio user name

conf - Alluxio configuration

Returns:

the group list of the user

Throws:

IOException

stripSuffixIfPresent

public static String stripSuffixIfPresent(String key,
                                          String suffix)

Strips the suffix if it exists. This method will leave keys without a suffix unaltered.

Parameters:

key - the key to strip the suffix from

suffix - suffix to remove

Returns:

the key with the suffix removed, or the key unaltered if the suffix is not present

stripPrefixIfPresent

public static String stripPrefixIfPresent(String key,
                                          String prefix)

Strips the prefix from the key if it is present. For example, for input key ufs://my-bucket-name/my-key/file and prefix ufs://my-bucket-name/, the output would be my-key/file. This method will leave keys without a prefix unaltered, ie. my-key/file returns my-key/file.

Parameters:

key - the key to strip

prefix - prefix to remove

Returns:

the key without the prefix

stripLeadingAndTrailingQuotes

public static String stripLeadingAndTrailingQuotes(String str)

Strips the leading and trailing quotes from the given string. E.g. return 'alluxio' for input '"alluxio"'.

Parameters:

str - The string to strip

Returns:

The string without the leading and trailing quotes

getValueFromStaticMapping

public static String getValueFromStaticMapping(String mapping,
                                               String key)

Gets the value with a given key from a static key/value mapping in string format. E.g. with mapping "id1=user1;id2=user2", it returns "user1" with key "id1". It returns null if the given key does not exist in the mapping.

Parameters:

mapping - the "key=value" mapping in string format separated by ";"

key - the key to query

Returns:

the mapped value if the key exists, otherwise returns ""

getRootCause

public static Throwable getRootCause(Throwable e)

Gets the root cause of an exception. It stops at encountering gRPC's StatusRuntimeException.

Parameters:

e - the exception

Returns:

the root cause

castToIOException

public static IOException castToIOException(Throwable e)

Casts a Throwable to an IOException.

Parameters:

e - the throwable

Returns:

the IO exception

singleElementIterator

public static <T> Iterator<T> singleElementIterator(T element)

Returns an iterator that iterates on a single element.

Type Parameters:

T - the type of the element

Parameters:

element - the element

Returns:

the iterator

invokeAll

public static <T> void invokeAll(List<Callable<T>> callables,
                                 long timeoutMs)
                          throws TimeoutException,
                                 ExecutionException

Executes the given callables, waiting for them to complete (or time out). If a callable throws an exception, that exception will be re-thrown from this method.

Type Parameters:

T - the return type of the callables

Parameters:

callables - the callables to execute

timeoutMs - time to wait for the callables to complete, in milliseconds

Throws:

TimeoutException - if the callables don't complete before the timeout

ExecutionException - if any of the callables throws an exception

closeAndRethrow

public static RuntimeException closeAndRethrow(com.google.common.io.Closer closer,
                                               Throwable t)
                                        throws IOException

Closes the Closer and re-throws the Throwable. Any exceptions thrown while closing the Closer will be added as suppressed exceptions to the Throwable. This method always throws the given Throwable, wrapping it in a RuntimeException if it's a non-IOException checked exception. Note: This method will wrap non-IOExceptions in RuntimeException. Do not use this method in methods which throw non-IOExceptions.

 Closer closer = new Closer();
 try {
   Closeable c = closer.register(new Closeable());
 } catch (Throwable t) {
   throw closeAndRethrow(closer, t);
 }
 

Parameters:

closer - the Closer to close

t - the Throwable to re-throw

Returns:

this method never returns

Throws:

IOException

unwrapResponse

public static void unwrapResponse(Protocol.Response response)
                           throws AlluxioStatusException

Unwraps a Protocol.Response.

Parameters:

response - the response

Throws:

AlluxioStatusException

unwrapResponseFrom

public static void unwrapResponseFrom(Protocol.Response response,
                                      io.netty.channel.Channel channel)
                               throws AlluxioStatusException

Unwraps a Protocol.Response associated with a channel.

Parameters:

response - the response

channel - the channel that receives this response

Throws:

AlluxioStatusException

isLocalHost

public static boolean isLocalHost(WorkerNetAddress address,
                                  AlluxioConfiguration conf)

Parameters:

address - the Alluxio worker network address

conf - Alluxio configuration

Returns:

true if the worker is local

convertMsToDate

public static String convertMsToDate(long millis,
                                     String dateFormatPattern)

Converts a millisecond number to a formatted date String.

Parameters:

millis - a long millisecond number

dateFormatPattern - the date format to follow when converting. i.e. mm-dd-yyyy

Returns:

formatted date String

convertMsToClockTime

public static String convertMsToClockTime(long millis)

Converts milliseconds to clock time.

Parameters:

millis - milliseconds

Returns:

input encoded as clock time

convertToByteString

public static Map<String,com.google.protobuf.ByteString> convertToByteString(Map<String,byte[]> input)

Parameters:

input - the input map

Returns:

a map using protobuf ByteString for values instead of byte[]

convertFromByteString

public static Map<String,byte[]> convertFromByteString(Map<String,com.google.protobuf.ByteString> input)

Parameters:

input - the input map

Returns:

a map using byte[] for values instead of protobuf ByteString

memoize

public static <T> java.util.function.Supplier<T> memoize(java.util.function.Supplier<T> original)

Memoize implementation for java.util.function.supplier.

Type Parameters:

T - the object type

Parameters:

original - the original supplier

Returns:

the supplier with memorization