Key-Value Adapter

Key-Value Stores

Most store implementations for the NStream persistence framework assume that the storage behaves like a cache. As changes are made to lanes in the application, they are passed directly into the underlying store. Typically in these cases, the underlying store takes care of buffering the write to disk. The Apache Ignite store implementation works in this way.

However, for some store implementations, particularly key-value stores like RocksDB, this would be inefficient. To support these types of stores, it is necessary to use an adapter.

To enable the adapter, add the nstream.persist.kv module to your module path and configure the store as follows:

storeName: @store {
    implName: "KeyValueAdapter"
    parameters: {
        implName: StoreImpl
        # Configuration paramteters for the key-value adapter go here.
        parameters: {
            # Configuration paramteters for the underlying store go here.
        }
    }
}

Here StoreImpl is the name of the underlying store implementation you want to use (e.g. RocksDB).

The key-value adapter will collect writes in memory and flush them periodically to the underlying store.

Configuration Options Table

The maximum commit interval is the maximum interval between commits. If this time elapses a commit will occur regardless of whether the minimum batch size has been reached. By default it is undefined meaning that the commit will not happen until the application stops if the batch size never exceeds the minimum.

When a commit does trigger, the data to be written is split into batches according to the maximum batch size. Each batch is written atomically. If a single value exceeds this limit, it will be written anyway to ensure progress.