(Experimental) Key Value System Client API

Slack Docker Pulls GitHub edit source

Overview

In addition to Filesystem API which allows applications to read, write or manage files, Alluxio also serves key-value system on top of Alluxio filesystem. Like files in Alluxio filesystem, the semantics of key-value system are also write-once:

  • Users can create a key-value store and insert key-value pairs into the store. A store becomes immutable after it is complete.
  • Users can open key-value stores after they are complete.

Each single key-value store is denoted by an AlluxioURI of the form alluxio://192.168.1.200:19998/path/my-kvstore, where the master address is 192.168.1.200, the PRC port is 19998, and the key-value store path is /path/my-kvstore. Depending on the total size and block size specified by the user, a single key-value store may consist of one or multiple partitions, but the internal is managed by Alluxio and thus transparent to users.

Configuration Parameters For Key-Value System

Key-Value support in Alluxio is disabled by default, and it can be enabled in Alluxio by setting alluxio.keyvalue.enabled to true (see configuration parameters)

Property NameDefaultMeaning
alluxio.keyvalue.enabledfalseWhether the key-value service is enabled.
alluxio.keyvalue.partition.size.bytes.max512MBMaximum allowable size of a single key-value partition in a store. This value should be no larger than the block size (alluxio.user.block.size.bytes.default).

Quick Test

After enabling Key-Value support in Alluxio, run ./bin/alluxio runKVTest to test whether the Key-Value system is running. It should output Passed the test! if the Key-Value system is correctly started.

Accessing Key-Value System in Java Application

Getting a Key-Value System Client

To obtain an Alluxio key-value system client in Java code, use:

  1. KeyValueSystem kvs = KeyValueSystem.Factory.create();

Creating a new key-value store

To create a new key-value store, use KeyValueSystem#createStore(AlluxioURI), which returns a writer to add key-value pairs. For example:

  1. KeyValueStoreWriter writer = kvs.createStore(new AlluxioURI("alluxio://192.168.1.200:19998/path/my-kvstore"));
  2. // Insert key-value pair ("100", "foo")
  3. writer.put("100", "foo");
  4. // Insert key-value pair ("200", "bar")
  5. writer.put("200", "bar");
  6. // Close and complete the store
  7. writer.close();

Note that:

  • Before the writer closes, the store is not complete and can not be read.
  • It is possible that the store is larger than the maximum allowed size of one partition. In this case, the writer will save key-value pairs into multiple partitions, but the switch is transparent.
  • The keys to insert should be sorted and with no duplicated keys.

Retrieving value from a store

To query a complete key-value store, use KeyValueSystem#openStore(AlluxioURI), which returns a reader to retrieve value by the key. For example:

  1. KeyValueStoreReader reader = kvs.openStore(new AlluxioURI("alluxio://192.168.1.200:19998/path/kvstore/"));
  2. // Return "foo"
  3. reader.get("100");
  4. // Return null as no value associated with "300"
  5. reader.get("300");
  6. // Close the reader on the store
  7. reader.close();

Iterating key-value pairs over a store

  1. KeyValueStoreReader reader = kvs.openStore(new AlluxioURI("alluxio://192.168.1.200:19998/path/kvstore/"));
  2. KeyValueIterator iterator = reader.iterator();
  3. while (iterator.hasNext()) {
  4.   KeyValuePair pair = iterator.next();
  5.   ByteBuffer key = pair.getkKey();
  6.   ByteBuffer value = pair.getValue();
  7. }
  8. // Close the reader on the store
  9. reader.close()

Examples

See more examples in the codebase.

Accessing Key-Value System in Hadoop MapReduce

MapReduce InputFormat

Alluxio provides an implementation of InputFormat for Hadoop MapReduce programs to access a key-value store. It takes a key-value URI, and emits key-value pairs stored in the store:

  1. conf.setInputFormat(KeyValueInputFormat.class);
  2. FileInputFormat.setInputPaths(conf, new Path("alluxio://192.168.1.200:19998/input-store"));

MapReduce OutputFormat

Similarly, Alluxio also provides implementations of OutputFormat and OutputCommitter for Hadoop MapReduce programs to create a key-value store by taking a key-value URI, and saving key-value pairs to the key-value store:

  1. conf.setOutputKeyClass(BytesWritable.class);
  2. conf.setOutputValueClass(BytesWritable.class);
  3. conf.setOutputFormat(KeyValueOutputFormat.class);
  4. FileOutputFormat.setOutputPath(conf, new Path("alluxio://192.168.1.200:19998/output-store"));

Examples

See an example in the codebase.

If you have configured Alluxio to use HDFS as under storage, and have enabled Key-Value system, you can run the example via

  1. export HADOOP_CLASSPATH=${ALLUXIO_INSTALLATION_DIRECTORY}/assembly/target/alluxio-assemblies-${ALLUXIO_VERSION}-jar-with-dependencies.jar
  3. ${HADOOP_INSTALLATION_DIRECTORY}/bin/hadoop jar \
  4.   ${ALLUXIO_INSTALLATION_DIRECTORY}/examples/target/alluxio-examples-${ALLUXIO_VERSION}.jar \
  5.   alluxio.examples.keyvalue.hadoop.CloneStoreMapReduce alluxio://${ALLUXIO_MASTER}:${PORT}/${INPUT_KEY_VALUE_STORE_PATH} alluxio://${ALLUXIO_MASTER}:${PORT}/${OUTPUT_KEY_VALUE_STORE_PATH} \
  6.   -libjars=${ALLUXIO_INSTALLATION_DIRECTORY}/assembly/target/alluxio-assemblies-${ALLUXIO_VERSION}-jar-with-dependencies.jar