Key Value System Client API (alpha)
- Overview
- Configuration Parameters For Key-Value System
- Quick Test
- Accessing Key-Value System in Java Application
- Accessing Key-Value System in Hadoop MapReduce
Overview
In addition to Filesystem API which allows applications to read, write or manage files, Alluxio also serves key-value system on top of Alluxio filesystem. Like files in Alluxio filesystem, the semantics of key-value system are also write-once:
- Users can create a key-value store and insert key-value pairs into the store. A store becomes immutable after it is complete.
- Users can open key-value stores after they are complete.
Each single key-value store is denoted by an AlluxioURI like alluxio://path/my-kvstore
.
Depending on the total size and block size specified by the user, a single key-value
store may consist of one or multiple partitions, but the internal is managed by Alluxio and thus
transparent to users.
Configuration Parameters For Key-Value System
Key-Value support in Alluxio is disabled by default, and it can be enabled in Alluxio by setting
alluxio.keyvalue.enabled
to true (see configuration parameters)
Property Name | Default | Meaning |
---|---|---|
alluxio.keyvalue.enabled | false | Whether the keyvalue interface is enabled. |
alluxio.keyvalue.partition.size.bytes.max | 512MB | Maximum size of each partition. |
Quick Test
After enabling Key-Value support in Alluxio, you can run ./bin/alluxio runKVTest
to test whether
the Key-Value system is running. You should see Passed the test!
at the bottom of the output if
the Key-Value system is correctly started.
Accessing Key-Value System in Java Application
Getting a Key-Value System Client
To obtain an Alluxio key-value system client in Java code, use:
KeyValueSystem kvs = KeyValueSystem.Factory.create();
Creating a new key-value store
To create a new key-value store, use KeyValueSystem#createStore(AlluxioURI)
, which returns
a writer to add key-value pairs. For example:
KeyValueStoreWriter writer = kvs.createStore(new AlluxioURI("alluxio://path/my-kvstore"));
// Insert key-value pair ("100", "foo")
writer.put("100", "foo");
// Insert key-value pair ("200", "bar")
writer.put("200", "bar");
// Close and complete the store
writer.close();
Note that,
- Before the writer closes, the store is not complete and can not be read;
- It is possible that the store is larger than the maximum allowed size of one partition, and in this case, the writer will save key-value pairs into multiple partitions. But the switch is transparent.
- The keys to insert should be sorted and with no duplicated keys.
Retrieving value from a store
To query a complete key-value store, use KeyValueSystem#openStore(AlluxioURI)
, which returns
a reader to retrieve value by the key. For example:
KeyValueStoreReader reader = kvs.openStore(new AlluxioURI("alluxio://path/kvstore/"));
// Return "foo"
reader.get("100");
// Return null as no value associated with "300"
reader.get("300");
// Close the reader on the store
reader.close();
Iterating key-value pairs over a store
KeyValueStoreReader reader = kvs.openStore(new AlluxioURI("alluxio://path/kvstore/"));
KeyValueIterator iterator = reader.iterator();
while (iterator.hasNext()) {
KeyValuePair pair = iterator.next();
ByteBuffer key = pair.getkKey();
ByteBuffer value = pair.getValue();
}
// Close the reader on the store
reader.close()
Examples
See more examples in the codebase.
Accessing Key-Value System in Hadoop MapReduce
MapReduce InputFormat
Alluxio provides an implementation of InputFormat
for Hadoop MapReduce programs to access
a key-value store. It takes a key-value URI, and emits key-value pairs stored in the store:
conf.setInputFormat(KeyValueInputFormat.class);
FileInputFormat.setInputPaths(conf, new Path("alluxio://input-store"));
MapReduce OutputFormat
Similarly, Alluxio also provides implementations of OutputFormat
and OutputCommitter
for Hadoop
MapReduce programs to create a key-value store by taking a key-value URI, and saving key-value
pairs to the key-value store:
conf.setOutputKeyClass(BytesWritable.class);
conf.setOutputValueClass(BytesWritable.class);
conf.setOutputFormat(KeyValueOutputFormat.class);
FileOutputFormat.setOutputPath(conf, new Path("alluxio://output-store"));
Examples
See an example in the codebase.
If you have configured Alluxio to use HDFS as under storage, and have enabled Key-Value system, you can run the example via
export HADOOP_CLASSPATH=${ALLUXIO_INSTALLATION_DIRECTORY}/assembly/target/alluxio-assemblies-${ALLUXIO_VERSION}-jar-with-dependencies.jar
${HADOOP_INSTALLATION_DIRECTORY}/bin/hadoop jar \
${ALLUXIO_INSTALLATION_DIRECTORY}/examples/target/alluxio-examples-${ALLUXIO_VERSION}.jar \
alluxio.examples.keyvalue.hadoop.CloneStoreMapReduce alluxio://${ALLUXIO_MASTER}:${PORT}/${INPUT_KEY_VALUE_STORE_PATH} alluxio://${ALLUXIO_MASTER}:${PORT}/${OUTPUT_KEY_VALUE_STORE_PATH} \
-libjars=${ALLUXIO_INSTALLATION_DIRECTORY}/assembly/target/alluxio-assemblies-${ALLUXIO_VERSION}-jar-with-dependencies.jar