Redis Streams

Redis Streams data ingestion is available using ingress of type redis.

Examples Repo

Example Redis Streams configuration files showcasing all features we will discuss can be found in our Examples Repo .

Settings

The Redis settings in the configuration file are used to configure data ingestion from Redis streams. Below is a detailed description of each field within the Redis settings:

Field	Required	Configurable	Description
`host`	`false`	`true`	Host name of the Redis instance to connect to.
`port`	`false`	`true`	Port number of the Redis instance to connect to.
`streams`	`true`	`true`	Streams field and starting ID, following the same format as the Redis CLI, e.g. `vehicles 0`.
`group`	`false`	`true`	The name of the consumer group.
`consumer`	`false`	`true`	The name of the consumer within the consumer group.
`block`	`false`	`true`	Time in ms to block for while waiting for new entries (default is to wait indefinitely).
`count`	`false`	`true`	The amount of entries to get with each read.
`user`	`false`	`true`	Redis user.
`password`	`false`	`true`	Redis password.
`database`	`false`	`true`	Redis database to connect to.
`clientName`	`false`	`true`	Client name to use for connection.
`ssl`	`false`	`true`	Whether to use SSL, `true` or `false`.
`caCertPath`	`false`	`true`	Path to truststore when using SSL.
`caCertPassword`	`false`	`true`	Password to truststore when using SSL.
`userCertPath`	`false`	`true`	Path to user certificate when using SSL.
`userCertPassword`	`false`	`true`	Password to user certificate when using SSL.
`timeoutMillis`	`false`	`true`	Connection and socket timeout in ms.
`connectionTimeoutMillis`	`false`	`true`	Connection timeout in ms.
`socketTimeoutMillis`	`false`	`true`	Socket timeout in ms.
`blockingSocketTimeoutMillis`	`false`	`true`	Blocking socket timeout in ms.
`maxTotal`	`false`	`true`	Maximum active connections in pool.
`maxIdle`	`false`	`true`	Maximum idle connections in pool.
`minIdle`	`false`	`true`	Minimum idle connections in pool.
`maxWait`	`false`	`true`	Maximum number of ms to wait for a connection to become available.
`timeBetweenEvictionRuns`	`false`	`true`	Time in ms between checking for idle connections in pool.
`blockWhenExhausted`	`false`	`true`	Block while waiting for connection to become available, `true` or `false`.
`testWhileIdle`	`false`	`true`	Enables sending a PING command periodically while the connection is idle, `true` or `false`.

Message Structure

The message structure of received Redis stream entries is:

{ "stream": ..., "id": ..., value: {...} }

Example

The configuration below defines a Redis Streams ingress where entries are consumed from the specified Redis stream and relayed to vehicle agents.

{
  "ingress": {
    "type": "redis",
    "settings": {
      "host": "host.docker.internal",
      "port": "6379",
      "streams": "vehicles 0"
    },
    "relay": {
      "agent": "vehicle",
      "idExtractor": "$value.id",
      "valueExtractor": "$value"
    }
  }
}

Pulsar

RocksDB Persistence

Nstream is licensed under the Redis Source Available License 2.0 (RSALv2).