Ingress Configuration

Ingress objects configure data sources in the ingress section of the configuration file. The ingress section accepts a single ingress object or an array of ingress objects. Here is a list of available fields for ingress objects:

The value of the type and settings fields are specific to each data source and will be covered in their corresponding sections.

Relays

Relay objects configure a single relay within an ingress object. The relay section accepts a single relay object or an array of relay objects. Here is a list of available fields for relay objects:

The simplest relay will use just agent and idExtractor fields; agent being the name or type of entity the data source is regarding, idExtractor being the extractor of unique ID from the received message.

In some cases you may not want to relay the whole message received. In this case you can pass a selector in the value field.

Let’s say the message received from a data source has the structure:

{"key": 111, "val": {"siteId":  111, status: "OK", ...}}

We can relay only the val field and forward that to the agent with the corresponding ID using:

"relay": {
  "agent": "site",
  "idExtractor": "$val.siteId", // "$key" would also work.
  "valueExtractor": "$val"      
}

Note: Each data source specific section will give the general structure of messages received to help configure extractors.

Complex and Composite URIs with node

Sometimes you may want to change the default node URI of agents, for instance if the unique ID of an entity is composed of two fields. In this case you can use the node field to pass a complex node URI.

Let’s extend the above example and say the message received has the structure:

{"key": 111, "val": {"siteId":  111, "zoneId": 222, status: "OK", ...}}

The site ID only being unique within a given zone ID. We can use the following relay to build a complex node URI and relay the val field to the corresponding site agent:

"relay": {
  "node": "/zone/$val.zoneId/site/$val.siteId",
  "valueExtractor": "$val"      
}

In this case we must ensure there is an agent with a matching node URI pattern in the agent section of the configuration file:

"agent": {
  "uri": "/zone/:zoneId/site/:siteId"
}

Looping Relays with forEach

The forEach field in the relay configuration is a powerful tool for processing arrays or collections within a single message. It acts as a selector that loops through each element of an array and relays each record individually to the specified agent. This is particularly useful when dealing with messages that contain multiple records bundled together, allowing each record to be handled separately and routed to the appropriate destination.

For example, if a message contains an array of events, using the forEach field ensures that each event is extracted and relayed individually:

"relay": {
  "agent": "event",
  "idExtractor": "$id",
  "forEach": "$value.events"
}

Certain ingress types will require the use of this feature as they return result sets - ingress specific pages will cover it.

Configurable Fields

Configurable fields in ingress settings allow developers to specify placeholders that will be replaced with actual values at runtime. They can be defined using placeholders enclosed within ${} syntax. There are two types of placeholders supported:

• Environment Variables: Placeholders of the form ${env.KEY} can be used to retrieve values from environment variables. Environment variables are commonly used to configure applications in various deployment environments.

• System Properties: Placeholders of the form ${prop.key} can be used to retrieve values from system properties. These values are typically provided as command-line arguments when launching the application.

Each data source section will highlight any fields that are configurable.

Here’s an example demonstrating the useage of configurable fields:

{
  "ingress": {
    "type": "kafka",
    "settings": {
      "topic": "${prop.kafka.topic}",
      "groupId": "${env.KAFKA_GROUP_ID}",
      ...
    }
  }
}

Properties Files

The properties field in the ingress configuration allows developers to specify an array of properties filenames. These properties files contain key-value pairs, where each key represents a setting, and the corresponding value provides the desired value for that setting. At runtime, Nstream-Jet can dynamically load these properties files and use the values to fill in any settings fields that are missing in the configuration. This enables developers to externalize configuration details into separate properties files, facilitating easier management and customization of application settings across different environments.

If all required settings are included in properties, then the settings field can be omitted.