Use Nstream to Model the Ever-Evolving State of Your Business

Fred Patton

How Nstream Built a Streaming App to Model Palo Alto Traffic

Streaming applications enable companies to fully harness their streaming data’s power to model the real-time state of their business continuously. This real-time visibility into the entire business – from functional operations to the entity (whether that entity be a truck, person, financial transaction, point-of-sale record, or other assets) – can help companies make more informed decisions, find efficiencies and cost savings, enhance automated actions, and even gain a competitive edge.

To see how you can build a streaming application, watch our demo video on building a streaming app to model real-time traffic in Palo Alto or read on below.

[caption id=”attachment_333” align=”alignleft” width=”499”] Click image to view Palo Alto traffic demo.[/caption]

A quick primer on streaming applications with Nstream

A streaming application is an application that is driven by real-time data and runs business logic immediately upon receiving data. With real-time streaming apps, business logic sits in the flow of your data by utilizing CPU and in-memory compute at speeds faster than the top speed of the network, so that your data doesn’t need to stop while being transformed. As a result, streaming apps built with Nstream can run at the speed of your fastest data sources.

Nstream reduces the complexity of traditional streaming architectures so you can build streaming applications in minutes, not months. With Nstream, you don’t need additional data systems, such as stream processing frameworks (e.g., ksqlDB, Flink, Kafka Streams, Pinot, Presto), additional databases (e.g., Druid, Clickhouse, MongoDB), additional applications servers (e.g., Spring, NodeJS, .NET), or data visualization software (e.g., Angular, React).

Nstream encompasses these capabilities while reducing tech stack complexity through three technology innovations:

• Stateful services: We call these Web Agents. Web Agents virtualize the state and behavior of every business entity, which can be a customer, prospect, sensor, and more. We flow data into Web Agents from multiple in-motion and at-rest data sources so they may respond with real-time business logic. Each Web Agent models the real-time state of its real-world counterpart and statefully computes on incoming data the instant it arrives. Web Agents also communicate with each other, sharing state and context to construct a global picture of the network from its component parts.
• Streaming APIs: These are built on top of Websockets that are extended and optimized for large-scale streaming of relational entities. Nstream streaming APIs provide granular, real-time updates for every Web Agent.
• Real-time User Interfaces (UIs): Real-time UIs enhance the real-time pipeline by including the browser, even though it sits outside of the processing network. Real-time UIs stream live data for just the Web Agents visible on screen. You can consume these same streaming APIs to incorporate real-time network data into your own applications.

Nstream’s SwimOS is the first open-source streaming application platform. Built on SwimOS, the Nstream Platform is the easiest and fastest way to configure, deploy, and manage streaming applications that unlock the full value of streaming data.

Visualizing real-time traffic in Palo Alto

We created a streaming application with the goal of observing real-time vehicle and pedestrian traffic in Palo Alto. Explore the Palo Alto streaming application for yourself and read on to see how we built it.

Building Web Agents to reflect Palo Alto’s traffic entities

Every application build starts the same way: by creating Web Agents. For each entity type that we model, we create a Web Agent to represent each instance in real-time.

For the city of Palo Alto, we created Web Agents for every intersection. These Web Agents are fueled with data from multiple in-motion and at-rest data sources, such as:

1. The intersection light: Designating either red, yellow, or green.
2. Detection sensors: Flagging when a vehicle passes through an intersection.
3. Call signals at crosswalks: Showing when pedestrians have hit the button for a chance to cross the street safely.
4. Static data about the intersection’s location and the lane’s schematics: These Web Agents combine with streaming data to create a more accurate picture.

The vast majority of sensor data is redundant and repeated at a high frequency, so having Web Agents at the edge that can process this data means that we can reduce a massive signal to a small, meaningful stream of information before propagating it through the network.

Because everything is streamed, sources can send once, and that transmission can be multicast to optimize network resources. At the consumer level, minimal state change deltas devoid of redundancy get multiplexed from each source into a straw instead of a firehose.

We also created a Web Agent for the city of Palo Alto itself. This allows us to see information such as:

• How many intersections have green lights, and whether traffic is flowing.
• The number of red lights.
• How many intersections have pedestrians waiting at crosswalks.

APIs ensure our map reflects real-time updates

Web Agents communicate with each other, sharing state and context and forming a complete map of your network from its component parts. This is made possible by streaming APIs. Streaming APIs ​​convey specific, contextualized information on a massive scale throughout your network, providing you with necessary context as soon as possible without asking for it.

After setting up all of our Web Agents in Palo Alto, we could then begin processing the incoming information. For this application, we processed 30,000 data points per second from the connected traffic intersections to create a complete picture of traffic in the area.

In this case, our network was broad, and with each Web Agent updating in real-time, we could see specific macro and micro landscape details: Users could zoom in and look at the vehicles and pedestrian traffic at an individual light, or they could zoom out to see what traffic looks like for the whole city.

Real-time UIs give your data value in real-time

Many UIs purport to be real-time, but the vast majority are not, as they push updates at set intervals or require manual refreshes. As a result, even though a company may process data in real-time, its UI may present a stale picture of its operations that further hampers time-sensitive decision-making efforts.

Nstream’s real-time UIs update instantaneously as entities change. That’s because Nstream’s real-time UIs connect objects via continuous streams that don’t compound latency by pausing for queues and database results –– even as users manipulate the data via the UI. In other words, when a user acts, the UI reacts in lockstep.

The real-time UI for the Palo Alto traffic app lets users better see, interpret, and then act on the incoming information. This might look like:

• Zooming in/out to see specific traffic lights or entire blocks of traffic.
• Quickly understanding where there’s vehicle and/or pedestrian traffic congestion.
• Automating business logic that changes the cadence of the city’s traffic lights once traffic hits a certain density to prevent traffic jams proactively.

How to get started with a streaming application

Streaming applications can be used to execute various common business use cases, no matter the industry. Here are some signs your organization may benefit from building and implementing a streaming application:

• Your current response time to changes in your environment hurts customer satisfaction.
• You lack visibility into your environment, which makes it difficult to make timely, data-backed decisions.
• Your current managed Kafka/Pulsar services are not delivering the true real-time experience you desire.
• Your current downstream analytics and decision automation are not keeping up with the top speed of your network.

Nstream is the fastest way to build streaming applications to model the real-time state of your business, whether you’re operating on cloud, on-prem, or both. If any of the above sounds like you, or you’d like to learn more about how a streaming application can support your business, get in touch.

For developers: Take a look at our in-depth tutorials and start building your own real-time streaming applications.