How we built the demo for the Current NOLA Day 2 keynote using Flink and AI
At Current 2025 in New Orleans this year we built a demo for the Day 2 keynote that would automagically summarise what was happening in the room, as …
Kafka to Iceberg - Exploring the Options
You’ve got data in Apache Kafka. You want to get that data into Apache Iceberg. What’s the best way to do it? Perhaps invariably, the answer is: IT …
Writing to Apache Iceberg on S3 using Kafka Connect with Glue catalog
Without wanting to mix my temperature metaphors, Iceberg is the new hawtness, and getting data into it from other places is a common task. I wrote …
It’s Time We Talked About Time: Exploring Watermarks (And More) In Flink SQL
Whether you’re processing data in batch or as a stream, the concept of time is an important part of accurate processing logic. Because we process …
Why is kcat showing the wrong topics?
Much as I love kcat (🤫 it’ll always be kafkacat to me…), this morning I nearly fell out with it 👇 😖 I thought I was going stir crazy, after listing …
Sending Data to Apache Iceberg from Apache Kafka with Apache Flink
This post originally appeared on the Decodable blog. Sometimes it’s not possible to have too much of a good thing, and whilst this blog may look at …
How to get data from Apache Kafka to Apache Iceberg on S3 with Decodable
This post originally appeared on the Decodable blog. Apache Iceberg is an open table format. It combines the benefits of data lakes (open standards, …
Using Apache Kafka with ngrok
Sometimes you might want to access Apache Kafka that’s running on your local machine from another device not on the same network. I’m not …
📌 🎁 A collection of Kafka-related talks 💝
Here’s a collection of Kafka-related talks, just for you. Each one has 🍿🎥 a recording, 📔 slides, and 👾 code to go and try out.
Flatten CDC records in KSQL
The problem - nested messages in Kafka Data comes into Kafka in many shapes and sizes. Sometimes it’s from CDC tools, and may be nested like …
Exploring JMX with jmxterm
Check out the jmxterm repository Download jmxterm from https://docs.cyclopsgroup.org/jmxterm
Accessing Kafka Docker containers’ JMX from host
See also docs. To help future Googlers… with the Confluent docker images for Kafka, KSQL, Kafka Connect, etc, if you want to access JMX metrics from …
Sending multiline messages to Kafka
(SO answer repost) You can use kafkacat to send messages to Kafka that include line breaks. To do this, use its -D operator to specify a custom …
Kafka Listeners - Explained
(This was cross-posted on the Confluent.io blog) This question comes up on StackOverflow and such places a lot, so here’s something to try and …
Analysing Network Data with Apache Kafka, KSQL, and Elasticsearch
In this article I demonstrated how to use KSQL to filter streams of network event data. As well as filtering, KSQL can be used to easily enrich …
Quick ’n Easy Population of Realistic Test Data into Kafka
tl;dr Use curl to pull data from the Mockaroo REST endpoint, and pipe it into kafkacat, thus: curl -s …
Streaming Data from MongoDB into Kafka with Kafka Connect and Debezium
Disclaimer: I am not a MongoDB person. These steps may or may not be appropriate and proper. But they worked for me :) Feel free to post in comments …
Streaming Data from MySQL into Kafka with Kafka Connect and Debezium
Debezium is a CDC tool that can stream changes from MySQL, MongoDB, and PostgreSQL into Kafka, using Kafka Connect. In this article we’ll see …
Streaming data from Kafka into Elasticsearch
This article is part of a series exploring Streaming ETL in practice. You can read about setting up the ingest of realtime events from a standard …
Installing the Python Kafka library from Confluent - troubleshooting some silly errors…
System: rmoff@proxmox01:~$ uname -a Linux proxmox01 4.4.6-1-pve #1 SMP Thu Apr 21 11:25:40 CEST 2016 x86_64 GNU/Linux rmoff@proxmox01:~$ head -n1 …