RocksDB Persistence

Nstream-Jet allows persistence using a RocksDB store.

Examples Repo

Example RocksDB configuration files showcasing all features we will discuss can be found in our Examples Repo .

To accept all default configuration and enable persistence with RocksDB quickly, set the value of the store field to RocksDB:

"app": {
  "store": "RocksDB"
}

Configuration

To configure the store, pass an object to the store field with structure:

"app": {
  "store": {
    "impl": "RocksDB",
    ... other configuration fields
  }
}

Here is a list of all available configuration fields:

Field	Default	Description
`path`	`/config/store`	Path in local filesystem where the database files will be located.
`writeBufferSize`	RocksDB default	Controls memory usage by write buffers and write-ahead log size.
`maxWalSize`	RocksDB default	Maximum size of the write-ahead log on disk.
`smallDb`	`false`	Flag to enable optimizations for small databases in RocksDB.
`minCommitInterval`	`60000`	Minimum time (in ms) between commits to the data store.
`maxCommitInterval`	`undefined`	Maximum time between commits to the data store (irrespective of `minCommitSize`).
`minCommitSize`	`4194304`	Minimum size (in bytes) of data before a commit is triggered.
`maxBatchSize`	`1048576`	Maximum size (in bytes) of each batch in a commit.
`shutdownTimeout`	`10000`	Time (in ms) allowed for the final commit to the database on server stop.

The maximum commit interval is the maximum interval between commits. If this time elapses a commit will occur regardless of whether the minimum batch size has been reached. By default, it is undefined meaning that the commit will not happen until the application stops if the batch size never exceeds the minimum.

When a commit does trigger, the data to be written is split into batches according to the maximum batch size. Each batch is written atomically. If a single value exceeds this limit, it will be written anyway to ensure progress.

Snapshots

By default, the RocksDB storing the state is kept in the local file system only. However, it is also possible to periodically copy snapshots to a remote repository. When the application is restarted, if a copy of the database is not available locally, it can be restored from the most recent snapshot. Currently, the only supported remote repository is S3 (or any other object store that uses the S3 protocol).

Field	Default	Description
`snapshotInterval`	-	How frequently the snapshots are taken (in ms).
`remoteSnapshotsRoot`	-	URI where remote snapshots will be stored.
`initialRetryDelay`	`0`	Initial time to wait (ms) before retrying a failed snapshot.
`maxRetry`	`0`	Maximum number of retry attempts (delay doubles each time from the initial value).
`keepHistory`	`0`	Number of RocksDB checkpoints to keep in the local filesystem.
`tag`	-	Arbitrary string that is appended to the root path to distinguish snapshot repositories for different servers within a single store.
`accessKey`	-	Access key to connect.
`secretKey`	-	Secret key to connect.
`region`	-	AWS region.
`endpoint`	AWS S3	URL of an S3 compatible store - leave absent for AWS S3.

snapshotInterval and remoteSnapshotsRoot are both required fields when using snapshots.

Example

The configuration below defines a RocksDB store that periodically snapshots the database to S3.

"store": {
  "impl": "RocksDB",
  "snapshotInterval": 120000,
  "remoteSnapshotsRoot": "s3://bucket_name/snapshots/",
  "tag": "nstream-1",
  "accessKey": "ACCESS_KEY",
  "secretKey": "SECRET_KEY",
  "region": "aws-global"        
}

Redis Streams

Nstream is licensed under the Redis Source Available License 2.0 (RSALv2).