March 7 - June 27, 2024 - Nstream is hitting the road with Confluent on the #DataInMotionTour! / Learn More

Key-Value Adapter

Key-Value Stores

Most store implementations for the NStream persistence framework assume that the storage behaves like a cache. As changes are made to lanes in the application, they are passed directly into the underlying store. Typically in these cases, the underlying store takes care of buffering the write to disk. The Apache Ignite store implementation works in this way.

However, for some store implementations, particularly key-value stores like RocksDB, this would be inefficient. To support these types of stores, it is necessary to use an adapter.

To enable the adapter, add the nstream.persist.kv module to your module path and configure the store as follows:

storeName: @store {
    implName: "KeyValueAdapter"
    parameters: {
        implName: StoreImpl
        # Configuration paramteters for the key-value adapter go here.
        parameters: {
            # Configuration paramteters for the underlying store go here.

Here StoreImpl is the name of the underlying store implementation you want to use (e.g. RocksDB).

The key-value adapter will collect writes in memory and flush them periodically to the underlying store.

Configuration Options Table

Parameter Default Value Description
minCommitInterval 60000 ms Minimum time between commits to the data store.
maxCommitInterval Undefined Maximum time between commits to the data store (irrespective of minCommitSize).
minCommitSize 4194304 bytes Minimum size of data before a commit is triggered.
maxBatchSize 1048576 bytes Maximum size of each batch in a commit.
shutdownTimeout 10000 ms Time allowed for the final commit to the database on server stop.

The maximum commit interval is the maximum interval between commits. If this time elapses a commit will occur regardless of whether the minimum batch size has been reached. By default it is undefined meaning that the commit will not happen until the application stops if the batch size never exceeds the minimum.

When a commit does trigger, the data to be written is split into batches according to the maximum batch size. Each batch is written atomically. If a single value exceeds this limit, it will be written anyway to ensure progress.

Nstream is licensed under the Redis Source Available License 2.0 (RSALv2).