A bash script to deploy ksqlDB queries automagically
There’s a bunch of improvements in the works for how ksqlDB handles code deployments and migrations. For now though, for deploying queries there’s …
Connecting to managed ksqlDB in Confluent Cloud with REST and ksqlDB CLI
Using ksqlDB in Confluent Cloud makes things a whole bunch easier because now you just get to build apps and streaming pipelines, instead of having …
Using ksqlDB to process data ingested from ActiveMQ with Kafka Connect
The ActiveMQ source connector creates a Struct holding the value of the message from ActiveMQ (as well as its key). This is as would be expected. …
Loading delimited data into Kafka - quick & dirty (but effective)
Whilst Apache Kafka is an event streaming platform designed for, well, streams of events, it’s perfectly valid to use it as a store of data which …
📼 ksqlDB HOWTO - A mini video series 📼
Some people learn through doing - and for that there’s a bunch of good ksqlDB tutorials here and here. Others may prefer to watch and listen first, …
Kafka Connect, ksqlDB, and Kafka Tombstone messages
As you may already realise, Kafka is not just a fancy message bus, or a pipe for big data. It’s an event streaming platform! If this is news to you, …
Streaming Geopoint data from Kafka to Elasticsearch
Streaming data from Kafka to Elasticsearch is easy with Kafka Connect - you can see how in this tutorial and video. One of the things that sometimes …
ksqlDB - How to model a variable number of fields in a nested value (STRUCT)
There was a good question on StackOverflow recently in which someone was struggling to find the appropriate ksqlDB DDL to model a source topic in …
📌 🎁 A collection of Kafka-related talks 💝
Here’s a collection of Kafka-related talks, just for you. Each one has 🍿🎥 a recording, 📔 slides, and 👾 code to go and try out.
Using the Debezium MS SQL connector with ksqlDB embedded Kafka Connect
Prompted by a question on StackOverflow I thought I’d take a quick look at setting up ksqlDB to ingest CDC events from Microsoft SQL Server using …
Counting the number of messages in a Kafka topic
There’s ways, and then there’s ways, to count the number of records/events/messages in a Kafka topic. Most of them are potentially inaccurate, or …
🤖Building a Telegram bot with Apache Kafka, Go, and ksqlDB
I had the pleasure of presenting at DataEngBytes recently, and am delighted to share with you the 🗒️ slides, 👾 code, and 🎥 recording of my ✨brand new …
Learning Golang (some rough notes) - S02E09 - Processing chunked responses before EOF is reached
The server sends Transfer-Encoding: chunked data, and you want to work with the data as you get it, instead of waiting for the server to finish, the …
Why JSON isn’t the same as JSON Schema in Kafka Connect converters and ksqlDB (Viewing Kafka messages bytes as hex)
I’ve been playing around with the new SerDes (serialisers/deserialisers) that shipped with Confluent Platform 5.5 - Protobuf, and JSON Schema (these …
Loading CSV data into Kafka
For whatever reason, CSV still exists as a ubiquitous data interchange format. It doesn’t get much simpler: chuck some plaintext with fields …
Working with JSON nested arrays in ksqlDB - example
Question from the Confluent Community Slack group: How can I access the data in object in an array like below using ksqlDB stream "Total": [ …
Building a Telegram bot with Apache Kafka and ksqlDB
Imagine you’ve got a stream of data; it’s not “big data,” but it’s certainly a lot. Within the data, you’ve got some bits you’re interested in, and …
Adventures in the Cloud, Part 94: ECS
My name’s Robin, and I’m a Developer Advocate. What that means in part is that I build a ton of demos, and Docker Compose is my jam. I love using …
Primitive Keys in ksqlDB
ksqlDB 0.7 will add support for message keys as primitive data types beyond just STRING (which is all we’ve had to date). That means that Kafka …
Monitoring Sonos with ksqlDB, InfluxDB, and Grafana
I’m quite a fan of Sonos audio equipment but recently had some trouble with some of the devices glitching and even cutting out whilst playing. Under …