The State of Streaming Data
The application layer will be the next domino to fall in the ongoing transformation from batch to streaming. And one particular quirk of streaming data—the distinction between state and change—will play an outsized role in determining the outcome.
The tradeoffs between state and change significantly impact the actionability of data products, data processing latency, and data transfer efficiency. Real-time streaming data applications can only scale—and only remain low latency—when they make effective use of both state and change data. As a rule of thumb, it’s easier to compute on state but more efficient to transmit change. Using change data where state is required can lead to a ten-million-fold slowdown in application responsiveness ( due to the latency of network round-trips needed to reconstruct state). At the same time, relying exclusively on state—where change data would suffice—can induce petabytes of excess data transfer (the cost of scraping datasets in search of change).
Nstream delivers the best of both worlds by computing on integrated state within Web Agents while transmitting derivative state changes over streaming APIs. This advancement opens the floodgates for streaming data to flow freely through the full application stack.
Racing to change state
To illustrate the differences between state and change data, consider the case of real-time retail inventory. An inventory system holds the current state of product inventory in a database. A complementary point-of-sale system tracks changes to product inventory as goods are sold.
This is sensible. A point-of-sale terminal sees barcode scans; it can directly measure changes to product inventory. It would cause a whole host of problems if a checkout register instead tried to fetch the current stock of a scanned item, subtract one, and write the new quantity back to the inventory database. If two different registers scanned the same UPC code for a jar of peanut butter at the same time, both terminals could observe that there are ten jars of peanut butter in stock, from which they would each subtract one, and then both update the inventory database to reflect that nine jars of peanut butter remain in stock. Two jars of peanut butter were sold, but the quantity in stock was only decreased by one. This is called a data race, and it’s a classic computer science quandary.
Somebody eventually has to update the inventory database though, right? Generating change data from point-of-sale terminals doesn’t eliminate the problem; it punts it somewhere else.
Relaying transactions
A common way to resolve data races is with transactions. In our point-of-sale example, a transaction could ensure that only one register at a time can scan, subtract, and update the stock of any given item. The downside of transactions is that they limit throughput. In our scenario, every time a register tried to scan an item, it would first have to verify that no other register was processing the same item, waiting for its turn to proceed when contended. The cost of these checks is incurred—and the latency of the checks accrued—regardless of how often registers actually step on each other’s toes.
In a more realistic scenario, an event processor consumes change-data-like messages generated by
point-of-sale terminals and asynchronously performs the necessary transactions to update the
inventory database. Checkout registers run without delay. And inventory databases stay eventually
consistent. Confluent robustly solves this problem with Kafka + Flink.
But there’s an important question we haven’t yet addressed: why do we even want a real-time
inventory system in the first place? What are the applications of real-time inventory? This is the
part of the race where Nstream picks up the baton.
Streaming to whom?
If the industry is so good at dealing with real-time inventory, why do half the items in my Instacart/DoorDash/Uber Eats orders always wind up out of stock? Although the data might be streaming to a retail inventory system, it’s not streaming to you.
Remember how we said at the outset that getting the balance between state and change wrong can lead to an explosion of cost and latency? Well, streaming real-time inventory data from the store all the way to you would break the bank, without something like Web Agents and Streaming APIs in the mix to make it efficient. Let’s examine why.
Stating the obvious
Do you like having to ask for the check at a restaurant? Or do you prefer when the server brings you the bill unprompted? It’s not just a convenience thing; it wastes everyone’s time if a customer asks for something repeatedly, only to hear back the same answer over and over again. On the other hand, the longer a customer waits to ask for what they need, the longer they may end up waiting to get it.
What if your server is incapable of handing over the check unless you request it? How often should
you inquire? Once a minute? Once a second? Every millisecond? To know in “real-time” when your check
is ready, you’d have to ask incessantly.
Most data-driven application architectures work like our least favorite restaurants. REST APIs—the
dominant means of exchanging application data—provide instantaneous snapshots of state, but only
on request. As a consequence, determining what data has changed often requires polling the entire
dataset. Scraping a store’s inventory system every second could cost millions of dollars a year just
for bandwidth. Since most product inventory doesn’t change second by second, the bulk of that
bandwidth would be wasted on redundant transfers of unchanged state. The inventory of some things
changes every second; the API just doesn’t say which things changed, putting the onus on the client
to compare whole snapshots to detect even a single changed byte.
Several layers of REST APIs sit between your food delivery app and the inventory databases of the stores you peruse. Each service polls the next one in a series at some undisclosed rate. Latencies accumulate, and accuracy deteriorates, leaving you to endure the crap user experience of not receiving what you ordered. Or, if you’re like me, you get fed up and revert to checking inventory the old-fashioned way—by driving to the store.
Fixing the problem
The problem can be generically solved by keeping track of precisely what changes as data gets enriched, transformed, and reshaped by the application layer. But tracking the state of all that change leads to untold numbers of read-modify-writes (i.e. data races) when attempted using traditional stateless application architectures. Transactions aren’t a scalable option due to the high rates of change that even a modest application undergoes as it computes, mashes up, and rolls up the hundreds—or even thousands—of derived states that can be altered by a single input event. Then there’s the question of how to efficiently inform clients about what changed. Do you tell them everything that changed? Or just the things they’re interested in that changed? And when do you tell clients what changed? Do they have to ask? Or can you stream incremental updates to them proactively? Nstream solves these exact problems as thoroughly and completely as Kafka + Flink solves the problem of reconciling point-of-sale data with inventory databases. Here’s a brief overview of how it works.
Nstream cascades state changes across a web of agents linked together by streaming APIs.
Business logic runs in stateful per-entity virtual compute processes called Web Agents. A Web Agent is like a Web Page that represents a process instead of a document. Web Agents circumvent the challenges of using change data by being stateful—there’s no state to reconstruct before handling a change because state is locally preserved between operations and continuously synchronized with related context.
Web Agents mitigate the trouble of dealing with state by transmitting precise change data to every streaming API client. When a Web Agent updates one of its states, it internally flags every API client currently observing that state as out-of-date. In parallel, when a connection to an API client is ready to send a packet, the platform transmits the highest priority delta between the current state and the known state of the client. All this happens automatically under the hood, without developers giving it much thought.
Applying the solution
Here’s how Nstream would make your food delivery app experience better. Every product line, in every store, gets its own dedicated Web Agent. The raison d’être for each of these product inventory agents is to track the stock of a single item, at a single store; and to make the inventory, cost, and other metadata about that particular product observable in real-time via streaming APIs. Product inventory agents can also track inventory history, project future availability, automate replenishment requests, and other entity-specific jobs. Connectors to the store’s inventory and/or point-of-sale systems forward messages about inventory changes to the particular Web Agents representing the products whose inventory changed.
Another class of Web Agents can be instantiated to aggregate the real-time inventory of whole
categories of products stocked at each store. Each category agent links to the streaming APIs of the
individual product inventory agents in its specific category at its specific store. This gives the
category agents a precise real-time view of all the inventory for a particular category without
having to redundantly subscribe to an overly broad broker topic. Category agents can also be tasked
with tracking common substitutions, correlating sales of similar items, and segmenting buyer
profiles.
The Web Agents themselves might run in-store on one or more edge devices, on a server cluster the
cloud, or distributed arbitrarily across edge and cloud. It makes little difference to the Web
Agents where they run; they can even be dynamically shuffled between hosts. A single moderately
sized VM or container can run millions of concurrent Web Agents; the fixed overhead per Web Agent is
only a few hundred bytes of memory.
Closing the loop
A mobile app can live data bind its UI view for a category of goods at a store directly to the streaming API of the corresponding category agent. Any changes to the store’s inventory received by the Web Agents will be reflected in the app with a latency of about 1/2 the round-trip time of the network—typically quite a bit faster than the blink of any eye. And the app will use a similar amount of bandwidth to serve you this real-time view as it currently uses to deliver stale results. Streaming APIs waste zero bytes polling for changes, so bandwidth use is proportional to how often the data actually changes instead of proportional to how often the app asks for changes.
It gets better though. A third party, such as a grocery delivery service, can stand up their own Web Agents for e.g. shopping carts, drivers, and customers. The Web Agent representing a customer’s shopping cart can link to the inventory agents of the products currently in its cart, even if the inventory agents and shopping carts agents are provided by different vendors and run on entirely separate infrastructure. If an item runs out of stock, the inventory agent immediately updates the shopping cart agent, and the shopping cart agent runs arbitrary business logic in response to the change, with full real-time context available to it at memory latency. The shopping cart agent could prompt the customer to choose a substitution, autonomously pick an alternative, delay or cancel the order, whatever the application developer wants it to do. Importantly, the business logic is proactively run in response to the real-time state change rather than on request. And the location of network partitions is transparent to the application.
Even better still, a shopping cart could monitor the real-time inventory of the specific products in the customer’s cart at multiple nearby stores. The shopping cart agent could continuously score which store is the best option for fulfilling the order, accounting for real-time inventory, drive time, cost of goods, weighting by item importance, etc… If a store runs out of the one thing the customer really wants while the shopper is on their way, the Web Agent could redirect the driver on-the-fly to the next best store. All this can be accomplished with trivial object-oriented composition, despite spanning multiple applications, distributed across partners and vendors, and running at the minimum latency of the network. And it will use less compute and bandwidth than the current high latency approach because of the granularity and precision innate to a stateful world wide web with first-class streaming APIs.
Learning more
These are just some of the ways Web Agents can be employed to enable real-time streaming data applications. Web Agent architectures can do so much more, with much less time investment, development effort, and operational cost than you might imagine. Check out SwimOS, the first full-stack streaming application platform, to start building your own streaming applications. And reach out to Nstream for low-code application templates, enterprise-grade tooling, and expert support.
