March 7 - June 27, 2024 - Nstream is hitting the road with Confluent on the #DataInMotionTour! / Learn More

Confluent Cloud Tutorial

Confluent Logo

The Kafka Vehicle Tutorial demonstrated how to build an Nstream application against a user-managed Kafka broker. In this tutorial, we recreate the same application against a SaaS managed broker, namely Confluent Cloud.

Recall that the Kafka tutorial:

This tutorial further demonstrates how to:

A standalone project containing all the discussed code can be found and cloned from here.

Nstream Library Dependencies

Dependencies

Gradle

implementation 'io.nstream:nstream-adapter-common:4.9.16'
implementation 'io.nstream:nstream-adapter-confluent:4.9.16'
implementation 'io.nstream:nstream-adapter-runtime:4.9.16'

Maven

<dependency>
  <groupId>io.nstream</groupId>
  <artifactId>nstream-adapter-common</artifactId>
  <version>4.9.16</version>
  <type>module</type>
</dependency>
<dependency>
  <groupId>io.nstream</groupId>
  <artifactId>nstream-adapter-confluent</artifactId>
  <version>4.9.16</version>
  <type>module</type>
</dependency>
<dependency>
  <groupId>io.nstream</groupId>
  <artifactId>nstream-adapter-runtime</artifactId>
  <version>4.9.16</version>
  <type>module</type>
</dependency>

Differences From Plain Kafka

The main, JSON-valued version of this application mimics the Kafka Vehicle Tutorial very closely (down to concrete class names), so we will not repeat any business logic explanations here. You will be able to follow along as long as you keep the following differences in mind:

Schema Registry Variation

Using a Schema Registry enables:

and other useful features.

If we use Avro, Confluent Cloud requires the following:

All of these may be supplied directly into the consumer.properties configuration as follows:

# consumer.properties

bootstrap.servers=pkc-rgm37.us-west-2.aws.confluent.cloud:9092
key.deserializer=org.apache.kafka.common.serialization.IntegerDeserializer
auto.offset.reset=latest
group.id=fooGroup

# ccloud
ccloud.api.key=FIXME: will resemble ALK1LKJ124LKAJD3
ccloud.api.secret=FIXME: will resemble BLahz+AKJHdjaAuexA/SnqXD+AKJHdjaAuex/aksjdn1927AAASJDNnadf121uB8

value.deserializer=io.confluent.kafka.serializers.KafkaAvroDeserializer
schema.registry.url=FIXME: will resemble https://abcd-ab1a1.us-east-2.aws.confluent.cloud
basic.auth.credentials.source=USER_INFO

# In the below property, FIXME1 and FIXME2 will resemble the ccloud properties above
#   (and may possibly, but usually will not, be the same; it is good practice to
#   have a separate Confluent Cloud API key for the Schema Registry).
basic.auth.user.info=FIXME1:FIXME2

That’s really it! You are free to use seamlessly use no-code or low-code ConfluentIngestingPatch variants as you wish.

Final Notes

Congratulations on building the backend for an end-to-end streaming application against SaaS-managed data! You may verify your progress with the same general purpose UI and swim-cli commands from the Kafka Vehicle Tutorial, linked here.


Nstream is licensed under the Redis Source Available License 2.0 (RSALv2).