It’s Time We Talked About Time: Exploring Watermarks (And More) In Flink SQL

Whether you’re processing data in batch or as a stream, the concept of time is an important part of accurate processing logic.

Because we process data after it happens, there are a minimum of two different types of time to consider:

When it happened, known as Event Time
When we process it, known as Processing Time (or system time or wall clock time)

Mar 13, 2025

Why is kcat showing the wrong topics?

Much as I love kcat (🤫 it’ll always be kafkacat to me…), this morning I nearly fell out with it 👇

😖 I thought I was going stir crazy, after listing topics on a broker and seeing topics from a different broker.

😵 WTF 😵

Nov 1, 2023

Using Apache Kafka with ngrok

Sometimes you might want to access Apache Kafka that’s running on your local machine from another device not on the same network. I’m not sure I can think of a production use-case, but there are a dozen examples for sandbox, demo, and playground environments.

In this post we’ll see how you can use ngrok to, in their words, Put localhost on the internet. And specifically, your local Kafka broker on the internet.

Sep 23, 2020

📌 🎁 A collection of Kafka-related talks 💝

Here’s a collection of Kafka-related talks, just for you.

Each one has 🍿🎥 a recording, 📔 slides, and 👾 code to go and try out.

Oct 11, 2018

Flatten CDC records in KSQL

The problem - nested messages in Kafka

Data comes into Kafka in many shapes and sizes. Sometimes it’s from CDC tools, and may be nested like this:

Sep 19, 2018

Exploring JMX with jmxterm

Check out the jmxterm repository
Download jmxterm from https://docs.cyclopsgroup.org/jmxterm

Sep 17, 2018

Accessing Kafka Docker containers’ JMX from host

Sep 4, 2018

Sending multiline messages to Kafka

(SO answer repost) You can use kafkacat to send messages to Kafka that include line breaks. To do this, use its -D operator to specify a custom message delimiter (in this example /): kafkacat -b kafka:29092 \ -t test_topic_01 \ -D/ \ -P <<EOF this is a string message with a line break/this is another message with two line breaks! EOF Note that the delimiter must be a single byte - multi-byte chars will end up getting included in the resulting message See issue #140

Aug 2, 2018

Kafka Listeners - Explained

(This was cross-posted on the Confluent.io blog)

This question comes up on StackOverflow and such places a lot, so here’s something to try and help.

tl;dr : You need to set advertised.listeners (or KAFKA_ADVERTISED_LISTENERS if you’re using Docker images) to the external address (host/IP) so that clients can correctly connect to it. Otherwise they’ll try to connect to the internal host address–and if that’s not reachable then problems ensue.

Put another way, courtesy of Spencer Ruport:

LISTENERS are what interfaces Kafka binds to. ADVERTISED_LISTENERS are how clients can connect.

Jun 17, 2018

Analysing Network Data with Apache Kafka, KSQL, and Elasticsearch

In this article I demonstrated how to use KSQL to filter streams of network event data. As well as filtering, KSQL can be used to easily enrich streams. In this article we’ll see how this enriched data can be used to drive analysis in Elasticsearch and Kibana—and how KSQL again came into use for building some stream processing as a result of the discovery made. The data came from my home Ubiquiti router, and took two forms: