alluxio.master.table.transform

Class TransformManager

java.lang.Object

alluxio.master.table.transform.TransformManager

All Implemented Interfaces:

Checkpointed, DelegatingJournaled, Journaled, JournalEntryIterable

public class TransformManager
extends Object
implements DelegatingJournaled

Manages transformations. It executes a transformation by submitting a job to the job service. A background thread periodically checks whether the job succeeds, fails, or is still running, the period is configurable by PropertyKey.TABLE_TRANSFORM_MANAGER_JOB_MONITOR_INTERVAL. It keeps information about all running transformations not only in memory, but also in the journal,so even if it is restarted, the information of the previous running transformations is not lost. It keeps history of all succeeded or failed transformations in memory for a time period which is configurable by PropertyKey.TABLE_TRANSFORM_MANAGER_JOB_HISTORY_RETENTION_TIME. If the job succeeds, it updates the Partition's location by AlluxioCatalog.completeTransformTable(JournalContext, String, String, String, Map).

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	TransformManager.ThrowingSupplier<R,E extends Throwable> A supplier with return type R that might throw exception E.

Constructor Summary

Constructors

Constructor and Description
TransformManager(TransformManager.ThrowingSupplier<JournalContext,UnavailableException> createJournalContext, AlluxioCatalog catalog, JobMasterClient jobMasterClient) An internal job master client will be created.

Method Summary

Modifier and Type	Method and Description
long	execute(String dbName, String tableName, TransformDefinition definition) Executes the plans for the table transformation.
List<TransformJobInfo>	getAllTransformJobInfo()
Journaled	getDelegate()
Optional<TransformJobInfo>	getTransformJobInfo(long jobId)
void	start(ExecutorService executorService, UserState userState) Starts background heartbeats.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface alluxio.master.journal.DelegatingJournaled

applyAndJournal, getCheckpointName, getJournalEntryIterator, processJournalEntry, resetState, restoreFromCheckpoint, writeToCheckpoint

Constructor Detail

TransformManager

public TransformManager(TransformManager.ThrowingSupplier<JournalContext,UnavailableException> createJournalContext,
                        AlluxioCatalog catalog,
                        JobMasterClient jobMasterClient)

An internal job master client will be created.

Parameters:

createJournalContext - journal context creator

catalog - the table catalog

jobMasterClient - the job master client

Method Detail

start

public void start(ExecutorService executorService,
                  UserState userState)

Starts background heartbeats. The heartbeats are stopped when unchecked error happens inside the heartbeats, or the hearbeat threads are interrupted, or the provided executor service is stopped. This class will not stop the executor service.

Parameters:

executorService - the executor service for executing heartbeat threads

userState - the user state for the heartbeat

execute

public long execute(String dbName,
                    String tableName,
                    TransformDefinition definition)
             throws IOException

Executes the plans for the table transformation. This method executes a transformation job with typeCompositeConfig, the transformation job concurrently executes the plans, each plan has a list of jobs to be executed sequentially. This method triggers the execution of the transformation job asynchronously without waiting for it to finish. The returned job ID can be used to poll the job service for the status of this transformation.

Parameters:

dbName - the database name

tableName - the table name

definition - the parsed transformation definition

Returns:

the job ID for the transformation job

Throws:

IOException - when there is an ongoing transformation on the table, or the transformation job fails to be started, or all partitions of the table have been transformed with the same definition

getTransformJobInfo

public Optional<TransformJobInfo> getTransformJobInfo(long jobId)

Parameters:

jobId - the job ID

Returns:

the job information

getAllTransformJobInfo

public List<TransformJobInfo> getAllTransformJobInfo()

Returns:

all transformation jobs, including the running jobs and the kept finished jobs, sorted by job ID in increasing order

getDelegate

public Journaled getDelegate()

Specified by:

getDelegate in interface DelegatingJournaled

Returns:

the object to delegate to