Google PubSub

Google PubSub data ingestion is available using ingress of type pubsub.

Examples Repo

Example Google PubSub configuration files showcasing all features we will discuss can be found in our Examples Repo .

Settings

The Google PubSub settings in the configuration file are used to configure data ingestion from Google PubSub subscriptions. Below is a detailed description of each field within the PubSub settings:

Field	Required	Configurable	Description
`projectId`	`true`	`true`	Google PubSub project ID.
`subscriptionId`	`true`	`true`	Google PubSub subscription ID - this must already be created.
`pubSubEmulatorHost`	`false`	`true`	The hostname of the Google PubSub emulator - if using the emulator.
`contentType`	`false`	`true`	The content type of the value, accepted types: `json`, `octet-stream`, `protobuf` (default `plaintext`).
`contentEncoding`	`false`	`true`	If using `octet-stream` content type, the encoding, currently accepted value `base64`.
`protoFilePath`	`false`	`true`	If using `protobuf` content type, the path to the protobuf schema file.

Message Structure

The message structure of received PubSub messages is (the message is just converted into a RECON value):

{ ... }

Google Application Credentials

Google PubSub ingestion - when not using an emulator - requires credentials to authenticate. After Nstream-Jet has been run, a pubsub-credentials.json file will be placed in the output directory, the contents of this file must be replaced with credentials obtained from Google before running. For this reason, the run container (-r) flag should not be used with Google PubSub ingress.

Example

The configuration below defines a PubSub ingress where JSON messages are consumed from the specified PubSub subscription and relayed to the appropriate agents for processing.

{
  "type": "pubsub",
  "settings": {
    "projectId": "vehicle-app",
    "subscriptionId": "nstream-vehicle-sub",
    "contentType": "json"
  },
  "relay": {
    "agent": "vehicle",
    "idExtractor": "$id"
  }
}

Kafka

MQTT

Nstream is licensed under the Redis Source Available License 2.0 (RSALv2).