public class TransformManager extends Object implements DelegatingJournaled
PropertyKey.TABLE_TRANSFORM_MANAGER_JOB_MONITOR_INTERVAL
.
It keeps information about all running transformations not only in memory,
but also in the journal,so even if it is restarted, the information of the previous running
transformations is not lost.
It keeps history of all succeeded or failed transformations in memory for a time period which is
configurable by PropertyKey.TABLE_TRANSFORM_MANAGER_JOB_HISTORY_RETENTION_TIME
.
If the job succeeds, it updates the Partition
's location by
AlluxioCatalog.completeTransformTable(JournalContext, String, String, String, Map)
.Modifier and Type | Class and Description |
---|---|
static interface |
TransformManager.ThrowingSupplier<R,E extends Throwable>
A supplier with return type R that might throw exception E.
|
Constructor and Description |
---|
TransformManager(TransformManager.ThrowingSupplier<JournalContext,UnavailableException> createJournalContext,
AlluxioCatalog catalog,
JobMasterClient jobMasterClient)
An internal job master client will be created.
|
Modifier and Type | Method and Description |
---|---|
long |
execute(String dbName,
String tableName,
TransformDefinition definition)
Executes the plans for the table transformation.
|
List<TransformJobInfo> |
getAllTransformJobInfo() |
Journaled |
getDelegate() |
Optional<TransformJobInfo> |
getTransformJobInfo(long jobId) |
void |
start(ExecutorService executorService,
UserState userState)
Starts background heartbeats.
|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
applyAndJournal, getCheckpointName, getJournalEntryIterator, processJournalEntry, resetState, restoreFromCheckpoint, writeToCheckpoint
public TransformManager(TransformManager.ThrowingSupplier<JournalContext,UnavailableException> createJournalContext, AlluxioCatalog catalog, JobMasterClient jobMasterClient)
createJournalContext
- journal context creatorcatalog
- the table catalogjobMasterClient
- the job master clientpublic void start(ExecutorService executorService, UserState userState)
executorService
- the executor service for executing heartbeat threadsuserState
- the user state for the heartbeatpublic long execute(String dbName, String tableName, TransformDefinition definition) throws IOException
CompositeConfig
,
the transformation job concurrently executes the plans,
each plan has a list of jobs to be executed sequentially.
This method triggers the execution of the transformation job asynchronously without waiting
for it to finish. The returned job ID can be used to poll the job service for the status of
this transformation.dbName
- the database nametableName
- the table namedefinition
- the parsed transformation definitionIOException
- when there is an ongoing transformation on the table, or the transformation
job fails to be started, or all partitions of the table have been transformed with the same
definitionpublic Optional<TransformJobInfo> getTransformJobInfo(long jobId)
jobId
- the job IDpublic List<TransformJobInfo> getAllTransformJobInfo()
public Journaled getDelegate()
getDelegate
in interface DelegatingJournaled
Copyright © 2023. All Rights Reserved.