March 7 - June 27, 2024 - Nstream is hitting the road with Confluent on the #DataInMotionTour! / Learn More

Lego vs. Playdough Use Cases

Playdough can be molded into any conceivable shape. Lego is much more restricted in the forms it can take.

There’s nothing that Lego can do that Playdough can’t theoretically also do.

Wouldn’t a Playdough Millennium Falcon be even cooler than a Lego Millennium Falcon?

Of course not! Working with Playdough is a nightmare.

What Lego lacks in flexibility it makes up for in composition. Every piece works seamlessly with every other piece, making complex creations practical and attainable to all.

Feature creep is a Playdough problem

Composability is sidelined in favor of precisely fitting into the mold of the use case.

It’s great at first. But if you have kids, you know what Playdough becomes after it’s sat out for awhile: a brittle crusty mess.

Most use cases are a combination of relatively standard capabilities

The use case should end in the shape of a Lego brick

If you build your use cases as bricks, then you only have to solve each problem once, and you can keep re-using that brick.

Desktop operating systems use one brick type for nearly everything: they’re called files

This is what desktop apps compose together so well.

You can edit files in a word processor, and then email those files with an email client. Your word processor doesn’t need to implement a custom email client in order to be able to send email. The apps are compostable because they all work with files.

The Web has one brick type: HTTP resources

Compare this with the web. Nearly every web application incorporates some custom mechanism to send emails, i.e. to email an itinerary after booking a trip. This is combinatorial complexity. Every app has to implement every permutation of features.

But the Web was built for documents, not big data—and definitely not for streaming data

The cases have changed, but the technology had not.

What’s the brick for data applications?

DataBricks would have you believe it’s a Spark RDD or DataFrame.

Problem: data is inherently Playdough-ey

Data makes for terrible bricks! Data is messy, noisy, non-uniform, and constantly shape-shifting in its structure.

Data is also incredibly expense to move.

A gigabyte per second can cost $150k/month per hop. Better minimize the number of hops your data takes!

If you use data as your bricks, you’re going to pay that cost over and over and over again as data moves through each component of a use case.

Solution: move the apps to the data instead of vice versa

Applications are built out of objects, which are much mire brick-ey.

Application logic is also infinitesimally tiny compared to data. Moving the app is effectively free. If you do it right, you may only have to move data once into the app platform. This os only possible with an integrated application stack.

Running apps in real-time is even better. Process data as it arrives and use it to update state, then throw out the original data before incurring the cost of storage and multiple network round-trips.

Stave off technical debt, feature creep, and combinatorial complexity by using many small apps that do one thing well and uniformly compose together to solve more complex problems

This is only practical if you can move apps to where the data is.

Moving apps to data requires a new kind of brick

SwimOS has one fundamental brick type: Web Agents

Web Agents extend the World Wide Web to encompass actively communicating processes, instead of just passive documents.

Web Agents are URI-addressable, just like any other web resource.

Each Wen Agent own and locally manage the data for a particular application entity.

Web Agents can dynamically execute arbitrary programs, like itty bitty web-native virtual machines.

Instead of moving billions of rows of data, you run billions of instances of small, highly composable programs

You only need to move the code for each program once to each physical machine.

Now your use cases can flexibly compose, with massively improved performance, decreased cost, and really real-time responsiveness