March 7 - June 27, 2024 - Nstream is hitting the road with Confluent on the #DataInMotionTour! / Learn More

Persistence with RocksDB

Why Use NStream Persistence?

NStream persistence, particularly with RocksDB, is an essential feature for modern, stateful applications. It provides several key advantages:

  1. State Preservation and Recovery: Preserves application state, crucial for scenarios requiring restarts and state regain. Ensures minimal disruption during unplanned outages or maintenance.

  2. Extended Historical State Storage: Allows storing a more extended historical state compared to traditional in-memory storage, which is limited and volatile. Ideal for applications needing long-term historical data access.

  3. High Performance and Scalability: Integrating with RocksDB, it offers high performance and scalability, essential for mission-critical applications demanding fast recovery and efficient handling of large data volumes.

  4. Increased Cache Limits: Significantly increases cache limits beyond typical web agent constraints, allowing storage capacities up to 1 TB without compromising latency.

  5. Reliability and Rollback Capability: Ensures a high degree of reliability with easy rollback processes during outages, aiming for 99.99% uptime.

Setting up NStream Persistence

Configuring NStream persistence involves a few simple steps. The configuration parameters are placed in additional slots within your existing configuration file. These parameters are flexible and can be tailored to meet specific application needs.

  1. Prerequisites: Ensure the RocksDB store implementation is in your module path.
  2. Kernel Declaration: Add this declaration to your server.recon file:
    @kernel(class: "nstream.persist.RocksKernel")
    
  3. Fabric Configuration: Specify the store in the fabric tag:
    storeName: @store {
        path: "/path/to/store"
    }
    

Configuration Options Table

Parameter Default Value Description
minCommitInterval 60000 ms Minimum time between commits to the data store.
minCommitSize 4194304 bytes Minimum size of data before a commit is triggered.
maxBatchSize 1048576 bytes Maximum size of each batch in a commit.
shutdownTimeout 10000 ms Time allowed for the final commit to the database on server stop.
writeBufferSize - Controls memory usage by write buffers and write-ahead log size.
maxWalSize - Maximum size of the write-ahead log on disk.
smallDb - Flag to enable optimizations for small databases in RocksDB.

How Persistence Works

Data Storage Location

Data is stored in the RocksDB database files in the directory specified by /path/to/store.

State Restoration on Restart

On restart, the state is restored from the RocksDB data store. Data in lanes marked with @SwimLane is reloaded, while data in lanes marked with @SwimTransient is not persisted and not restored.

Lane Level Configuration

Example Application

Explore persistent-app, an example application using persistence, available at: GitHub - NStream Persistent App.


Nstream is licensed under the Redis Source Available License 2.0 (RSALv2).