At Current 2025 in New Orleans this year we built a demo for the Day 2 keynote that would automagically summarise what was happening in the room, as reported by members of the audience. Here’s how we did it!
You’ve got data in Apache Kafka.
You want to get that data into Apache Iceberg.
What’s the best way to do it?
Perhaps invariably, the answer is: IT DEPENDS. But fear not: here is a guide to help you navigate your way to choosing the best solution for you 🫵.
Without wanting to mix my temperature metaphors, Iceberg is the new hawtness, and getting data into it from other places is a common task. I wrote previously about using Flink SQL to do this, and today I’m going to look at doing the same using Kafka Connect.
Kafka Connect can send data to Iceberg from any Kafka topic. The source Kafka topic(s) can be populated by a Kafka Connect source connector (such as Debezium), or a regular application producing directly to it.
Whether you’re processing data in batch or as a stream, the concept of time is an important part of accurate processing logic.
Because we process data after it happens, there are a minimum of two different types of time to consider:
When it happened, known as Event Time
When we process it, known as Processing Time (or system time or wall clock time)
Much as I love kcat (🤫 it’ll always be kafkacat to me…), this morning I nearly fell out with it 👇
😖 I thought I was going stir crazy, after listing topics on a broker and seeing topics from a different broker.
😵 WTF 😵
|
Note
|
This post originally appeared on the Decodable blog. |
Sometimes it’s not possible to have too much of a good thing, and whilst this blog may look at first-glance rather similar to the one that I published just recently , today we’re looking at a 100% pure Apache solution. Because who knows, maybe you prefer rolling your own tech stacks instead of letting Decodable do it for you 😉.
|
Note
|
This post originally appeared on the Decodable blog. |
Apache Iceberg is an open table format. It combines the benefits of data lakes (open standards, cheap object storage) with the good things that data warehouses have, like first-class support for tables and SQL capabilities including updates to data in place, time-travel, and transactions. With the recent acquisition by Databricks of Tabular—one of the main companies that contribute to Iceberg—it’s clear that Iceberg is winning out as one of the primary contenders in this space.
Sometimes you might want to access Apache Kafka that’s running on your local machine from another device not on the same network. I’m not sure I can think of a production use-case, but there are a dozen examples for sandbox, demo, and playground environments.
In this post we’ll see how you can use ngrok to, in their words, Put localhost on the internet. And specifically, your local Kafka broker on the internet.
Here’s a collection of Kafka-related talks, just for you.
Each one has 🍿🎥 a recording, 📔 slides, and 👾 code to go and try out.
Data comes into Kafka in many shapes and sizes. Sometimes it’s from CDC tools, and may be nested like this:
See also docs.
To help future Googlers… with the Confluent docker images for Kafka, KSQL, Kafka Connect, etc, if you want to access JMX metrics from within, you just need to pass two environment variables: <x>_JMX_HOSTNAME and <x>_JMX_PORT, prefixed by a component name.
<x>_JMX_HOSTNAME - the hostname/IP of the JMX host machine, as accessible from the JMX Client.
This is used by the JMX client to connect back into JMX, so must be accessible from the host machine running the JMX client.
You can use kafkacat to send messages to Kafka that include line breaks. To do this, use its -D operator to specify a custom message delimiter (in this example /):
kafkacat -b kafka:29092 \
-t test_topic_01 \
-D/ \
-P <<EOF
this is a string message
with a line break/this is
another message with two
line breaks!
EOF
Note that the delimiter must be a single byte - multi-byte chars will end up getting included in the resulting message See issue #140
(This was cross-posted on the Confluent.io blog)
This question comes up on StackOverflow and such places a lot, so here’s something to try and help.
tl;dr : You need to set advertised.listeners (or KAFKA_ADVERTISED_LISTENERS if you’re using Docker images) to the external address (host/IP) so that clients can correctly connect to it. Otherwise they’ll try to connect to the internal host address–and if that’s not reachable then problems ensue.
Put another way, courtesy of Spencer Ruport:
LISTENERSare what interfaces Kafka binds to.ADVERTISED_LISTENERSare how clients can connect.
In this article I demonstrated how to use KSQL to filter streams of network event data. As well as filtering, KSQL can be used to easily enrich streams. In this article we’ll see how this enriched data can be used to drive analysis in Elasticsearch and Kibana—and how KSQL again came into use for building some stream processing as a result of the discovery made.
The data came from my home Ubiquiti router, and took two forms:
tl;dr Use curl to pull data from the Mockaroo REST endpoint, and pipe it into kafkacat, thus:
curl -s "https://api.mockaroo.com/api/d5a195e0?count=2&key=ff7856d0"| \
kafkacat -b localhost:9092 -t purchases -P
Disclaimer: I am not a MongoDB person. These steps may or may not be appropriate and proper. But they worked for me :) Feel free to post in comments if I’m doing something wrong
For Debezium to be able to stream changes from MongoDB, Mongo needs to have replication configured:
Docs: Replication / Convert a Standalone to a Replica Set
Stop Mongo:
rmoff@proxmox01 ~> sudo service mongod stop
Add replica set config to /etc/mongod.conf:
Debezium is a CDC tool that can stream changes from MySQL, MongoDB, and PostgreSQL into Kafka, using Kafka Connect. In this article we’ll see how to set it up and examine the format of the data. A subsequent article will show using this realtime stream of data from a RDBMS and join it to data originating from other sources, using KSQL.
The software versions used here are:
To use it, you need the relevant JAR for the source system (e.g. MySQL), and make that JAR available to Kafka Connect. Here we’ll set it up for MySQL.
This article is part of a series exploring Streaming ETL in practice. You can read about setting up the ingest of realtime events from a standard Oracle platform, and building streaming ETL using KSQL.
This post shows how we take data streaming in from an Oracle transactional system into Kafka, and simply stream it onwards into Elasticsearch. This is a common pattern, for enabling rapid search or analytics against data held in systems elsewhere.
System:
rmoff@proxmox01:~$ uname -a
Linux proxmox01 4.4.6-1-pve #1 SMP Thu Apr 21 11:25:40 CEST 2016 x86_64 GNU/Linux
rmoff@proxmox01:~$ head -n1 /etc/os-release
PRETTY_NAME="Debian GNU/Linux 8 (jessie)"
rmoff@proxmox01:~$ python --version
Python 2.7.9
Following:
Install librdkafka, which is a pre-req for the Python library:
wget -qO - https://packages.confluent.io/deb/4.0/archive.key | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://packages.confluent.io/deb/4.0 stable main"
sudo apt-get install librdkafka-dev python-dev
Setup virtualenv:
sudo apt-get install virtualenv
virtualenv kafka_push_notify
source ./kafka_push_notify/bin/activate.fish
Try to install confluent-kafka: