alluxio.hadoop.mapreduce

Class KeyValueOutputCommitter

java.lang.Object

org.apache.hadoop.mapreduce.OutputCommitter

org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter

alluxio.hadoop.mapreduce.KeyValueOutputCommitter

@PublicApi
 @ThreadSafe
public final class KeyValueOutputCommitter
extends org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter

Extension of FileOutputCommitter where creating, completing, or deleting a KeyValueSystem in different phases of a job's or task's lifecycle is considered.

This committer must be used along with KeyValueOutputFormat to merge the key-value stores created by each Reducer into one key-value store under the MapReduce output directory.

Field Summary

Fields inherited from class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter

PENDING_DIR_NAME, SUCCEEDED_FILE_NAME, SUCCESSFUL_JOB_OUTPUT_DIR_MARKER, TEMP_DIR_NAME

Constructor Summary

Constructors

Constructor and Description
KeyValueOutputCommitter(org.apache.hadoop.fs.Path outputPath, org.apache.hadoop.mapreduce.TaskAttemptContext taskContext) Constructor.

Method Summary

Modifier and Type	Method and Description
void	abortTask(org.apache.hadoop.mapreduce.TaskAttemptContext taskContext) Deletes the completed key-value stores under the task's temporary output directory, and then calls FileOutputCommitter.abortTask(TaskAttemptContext).
void	commitTask(org.apache.hadoop.mapreduce.TaskAttemptContext taskContext) Merges the completed key-value store under the task's temporary output directory to the key-value store at job output directory, then calls FileOutputCommitter.commitTask(TaskAttemptContext).
static String	getPendingDirName()
boolean	needsTaskCommit(org.apache.hadoop.mapreduce.TaskAttemptContext taskContext)

Methods inherited from class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter

abortJob, abortTask, cleanupJob, commitJob, commitTask, getCommittedTaskPath, getCommittedTaskPath, getCommittedTaskPath, getJobAttemptPath, getJobAttemptPath, getJobAttemptPath, getTaskAttemptPath, getTaskAttemptPath, getWorkPath, isRecoverySupported, needsTaskCommit, recoverTask, setupJob, setupTask

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

KeyValueOutputCommitter

public KeyValueOutputCommitter(org.apache.hadoop.fs.Path outputPath,
                               org.apache.hadoop.mapreduce.TaskAttemptContext taskContext)
                        throws IOException

Constructor.

Parameters:

outputPath - the job's output path, or null if the output committer is a noop

taskContext - the task's context

Throws:

IOException

Method Detail

needsTaskCommit

public boolean needsTaskCommit(org.apache.hadoop.mapreduce.TaskAttemptContext taskContext)
                        throws IOException

Overrides:

needsTaskCommit in class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter

Parameters:

taskContext - MapReduce task configuration

Returns:

true if the task output directory exists, otherwise false

Throws:

IOException

commitTask

public void commitTask(org.apache.hadoop.mapreduce.TaskAttemptContext taskContext)
                throws IOException

Merges the completed key-value store under the task's temporary output directory to the key-value store at job output directory, then calls FileOutputCommitter.commitTask(TaskAttemptContext).

Overrides:

commitTask in class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter

Throws:

IOException

abortTask

public void abortTask(org.apache.hadoop.mapreduce.TaskAttemptContext taskContext)

Deletes the completed key-value stores under the task's temporary output directory, and then calls FileOutputCommitter.abortTask(TaskAttemptContext).

Overrides:

abortTask in class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter

getPendingDirName

public static String getPendingDirName()

Returns:

the temp directory name