Configuration File

The configuration file for Nstream-Jet serves as the blueprint for generating Nstream applications. It defines the connection details for data sources and specifies the structure and behavior of the resulting application. The configuration file is written in JSON format, providing a clear and concise way to encapsulate the necessary parameters and settings.

The file follows a structured format, comprising distinct sections that define various aspects of the generated Nstream application. Here’s a breakdown of the key fields:

{
  "app": {...},
  "ingress": [...],
  "agent": [...]
}

app

The app section configures application-wide settings, such as the name of the application and the port on which the application will run. These settings define fundamental characteristics of the Nstream application.

The app field is optional, all values will default if omitted. Here is a list of available fields:

ingress

The ingress section focuses on configuring data sources that feed into the Nstream application. It specifies the type of data source and provides settings specific to that data source. Additionally, the relay sub-section defines how data is relayed within the application, including the agent responsible for processing the data.

The ingress field is required as at least one data source is needed to generate entities. The field accepts either a single ingress object or a list of ingress objects:

"ingress": [{...}, ..., {...}]    or    "ingress:" {...}

Ingress objects and specific data source configurations will be covered in the upcoming Ingress Configuration section.

agent

The agent section is dedicated to configuring entities and their behavior within the Nstream application. Here, developers can define settings for individual agents, such as their URIs and any patches. Patches allow developers to customize agent behavior, such as specifying retention count for historical data.

The agent field is optional, if omitted all agents will be inferred from relays and given some simple default behavior. The field accepts either a single agent object or a list of agent objects:

"agent": [{...}, ..., {...}]    or    "agent:" {...} 

Agent objects and adding behavior to entities will be covered in the upcoming Agent Configuration section.

Example

Here we give a simple example for configuring an application using a single data source. We will accept some defaults for simplicity, but we will show how to fully customise your application in following sections.

Let’s say we have a single Kafka data source with a topic vehicles streaming updates for a fleet of vehicles. We can build our configuration file as follows:

• First we will omit the app field, accepting the defaults.
• We will also omit the agent field, allowing for default behavior of our vehicle entities.
• Consulting the Kafka section we configure a single ingress with necessary settings.
• We add a single relay to the ingress, extracting the unique ID of the vehicle and relaying the message to the vehicle agent.

{
  "ingress": {
    "type": "kafka",
    "settings": {
      "bootstrapServers": "localhost",
      "topic": "vehicles",
      "groupId": "nstream-jet-app",
      "keyDeserializer":  "string",
      "valueDeserializer": "string"
    },
    "relay": {
      "agent": "vehicle",
      "idExtractor": "$vehicleId"
    }
  }
}

That’s it! We can now save this file and generate a streaming application using Nstream-Jet.