🎄 Twelve Days of SMT 🎄 - Day 2: ValueToKey and ExtractField
Setting the key of a Kafka message is important as it ensures correct logical processing when consumed across multiple partitions, as well as being a …
🎄 Twelve Days of SMT 🎄 - Day 1: InsertField (timestamp)
You can use the InsertField Single Message Transform (SMT) to add the message timestamp into each message that Kafka Connect sends to a sink. To use …
Kafka Connect, ksqlDB, and Kafka Tombstone messages
As you may already realise, Kafka is not just a fancy message bus, or a pipe for big data. It’s an event streaming platform! If this is news to you, …
Streaming Geopoint data from Kafka to Elasticsearch
Streaming data from Kafka to Elasticsearch is easy with Kafka Connect - you can see how in this tutorial and video. One of the things that sometimes …
Streaming XML messages from IBM MQ into Kafka into MongoDB
Let’s imagine we have XML data on a queue in IBM MQ, and we want to ingest it into Kafka to then use downstream, perhaps in an application or maybe …
Ingesting XML data into Kafka - Option 3: Kafka Connect FilePulse connector
👉 Ingesting XML data into Kafka - Introduction We saw in the first post how to hack together an ingestion pipeline for XML into Kafka using a source …
Ingesting XML data into Kafka - Option 2: Kafka Connect plus Single Message Transform
We previously looked at the background to getting XML into Kafka, and potentially how [not] to do it. Now let’s look at the proper way to build a …
What is Kafka Connect?
Kafka Connect is the integration API for Apache Kafka. Check out this video for an overview of what Kafka Connect enables you to do, and how to do it. …
How to install connector plugins in Kafka Connect
Kafka Connect (which is part of Apache Kafka) supports pluggable connectors, enabling you to stream data between Kafka and numerous types of system, …
Loading CSV data into Kafka
For whatever reason, CSV still exists as a ubiquitous data interchange format. It doesn’t get much simpler: chuck some plaintext with fields …
Kafka Connect JDBC Sink - setting the key field name
I wanted to get some data from a Kafka topic: ksql> PRINT PERSON_STATS FROM BEGINNING; Key format: KAFKA (STRING) Value format: AVRO rowtime: …
Primitive Keys in ksqlDB
ksqlDB 0.7 will add support for message keys as primitive data types beyond just STRING (which is all we’ve had to date). That means that Kafka …
Notes on getting data into InfluxDB from Kafka with Kafka Connect
You can download the InfluxDB connector for Kafka Connect here. Documentation for it is here. When a message from your source Kafka topic is written …
Kafka Connect and Schemas
Here’s a fun one that Kafka Connect can sometimes throw out: java.lang.ClassCastException: java.util.HashMap cannot be cast to …
Monitoring Sonos with ksqlDB, InfluxDB, and Grafana
I’m quite a fan of Sonos audio equipment but recently had some trouble with some of the devices glitching and even cutting out whilst playing. Under …
Changing the Logging Level for Kafka Connect Dynamically
Logs are magical things. They tell us what an application is doing—or not doing. They help us debug problems. As it happens, they also underpin the …
Streaming messages from RabbitMQ into Kafka with Kafka Connect
This was prompted by a question on StackOverflow to which I thought the answer would be straightforward, but turned out not to be so. And then I got …
Analysing network behaviour with ksqlDB and MongoDB
In this post I want to build on my previous one and show another use of the Syslog data that I’m capturing. Instead of looking for SSH attacks, I’m …
Detecting and Analysing SSH Attacks with ksqlDB
I’ve written previously about ingesting Syslog into Kafka and using KSQL to analyse it. I want to revisit the subject since it’s nearly two years …
Kafka Connect - Request timed out
A short & sweet blog post to help people Googling for this error, and me next time I encounter it. The scenario: trying to create a connector in …