alluxio.hadoop.mapreduce

Class KeyValueOutputFormat

java.lang.Object

org.apache.hadoop.mapreduce.OutputFormat<K,V>

org.apache.hadoop.mapreduce.lib.output.FileOutputFormat<org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable>

alluxio.hadoop.mapreduce.KeyValueOutputFormat

@PublicApi
 @ThreadSafe
public final class KeyValueOutputFormat
extends org.apache.hadoop.mapreduce.lib.output.FileOutputFormat<org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable>

An OutputFormat implementation to let MapReduce job write to a key-value store under the MapReduce output directory.

There are different key-value stores under the output directory configured via MapReduce property mapreduce.output.fileoutputformat.outputdir for different tasks. The stores are merged into one key-value store under the MapReduce output directory by KeyValueOutputCommitter.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.mapreduce.lib.output.FileOutputFormat

org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.Counter

Field Summary

Fields inherited from class org.apache.hadoop.mapreduce.lib.output.FileOutputFormat

BASE_OUTPUT_NAME, COMPRESS, COMPRESS_CODEC, COMPRESS_TYPE, OUTDIR, PART

Constructor Summary

Constructors

Constructor and Description
KeyValueOutputFormat() Constructs a new KeyValueOutputFormat.

Method Summary

Modifier and Type	Method and Description
void	checkOutputSpecs(org.apache.hadoop.mapreduce.JobContext jobContext)
static AlluxioURI	getJobOutputURI(org.apache.hadoop.mapreduce.JobContext jobContext)
org.apache.hadoop.mapreduce.OutputCommitter	getOutputCommitter(org.apache.hadoop.mapreduce.TaskAttemptContext taskContext)
org.apache.hadoop.mapreduce.RecordWriter<org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable>	getRecordWriter(org.apache.hadoop.mapreduce.TaskAttemptContext taskAttemptContext)
static AlluxioURI	getTaskOutputURI(org.apache.hadoop.mapreduce.TaskAttemptContext taskContext)

Methods inherited from class org.apache.hadoop.mapreduce.lib.output.FileOutputFormat

getCompressOutput, getDefaultWorkFile, getOutputCompressorClass, getOutputName, getOutputPath, getPathForWorkFile, getUniqueFile, getWorkOutputPath, setCompressOutput, setOutputCompressorClass, setOutputName, setOutputPath

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

KeyValueOutputFormat

public KeyValueOutputFormat()

Constructs a new KeyValueOutputFormat.

Method Detail

getTaskOutputURI

public static AlluxioURI getTaskOutputURI(org.apache.hadoop.mapreduce.TaskAttemptContext taskContext)

Parameters:

taskContext - MapReduce task context

Returns:

the task's temporary output path ${job output directory}/_temporary/${task attempt id}

getJobOutputURI

public static AlluxioURI getJobOutputURI(org.apache.hadoop.mapreduce.JobContext jobContext)

Parameters:

jobContext - MapReduce job configuration

Returns:

the job's output path

getRecordWriter

public org.apache.hadoop.mapreduce.RecordWriter<org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable> getRecordWriter(org.apache.hadoop.mapreduce.TaskAttemptContext taskAttemptContext)
                                                                                                                                throws IOException

Specified by:

getRecordWriter in class org.apache.hadoop.mapreduce.lib.output.FileOutputFormat<org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable>

Throws:

IOException

checkOutputSpecs

public void checkOutputSpecs(org.apache.hadoop.mapreduce.JobContext jobContext)
                      throws IOException

This method is called immediately when job is submitted, a key-value store is created at the job's output directory, key-value stores created by MapReduce tasks will be merged into this store when task is submitted.

Overrides:

checkOutputSpecs in class org.apache.hadoop.mapreduce.lib.output.FileOutputFormat<org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable>

Throws:

IOException

getOutputCommitter

public org.apache.hadoop.mapreduce.OutputCommitter getOutputCommitter(org.apache.hadoop.mapreduce.TaskAttemptContext taskContext)
                                                               throws IOException

Overrides:

getOutputCommitter in class org.apache.hadoop.mapreduce.lib.output.FileOutputFormat<org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable>

Parameters:

taskContext - MapReduce task configuration

Returns:

a KeyValueOutputCommitter

Throws:

IOException