Creating an HTTP Source connector on Confluent Cloud from the CLI
In this blog article I’ll show you how you can use the confluent CLI to set up a Kafka cluster on Confluent Cloud, the necessary API keys, and then a …
Why is kcat showing the wrong topics?
Much as I love kcat (🤫 it’ll always be kafkacat to me…), this morning I nearly fell out with it 👇 😖 I thought I was going stir crazy, after listing …
Quick profiling of data in Apache Kafka using kafkacat and visidata
ksqlDB is a fantastically powerful tool for processing and analysing streams of data in Apache Kafka. But sometimes, you just want a quick way to …
Loading delimited data into Kafka - quick & dirty (but effective)
Whilst Apache Kafka is an event streaming platform designed for, well, streams of events, it’s perfectly valid to use it as a store of data which …
Performing a GROUP BY on data in bash
One of the fun things about working with data over the years is learning how to use the tools of the day—but also learning to fall back on the tools …
Ingesting XML data into Kafka - Option 1: The Dirty Hack
👉 Ingesting XML data into Kafka - Introduction What would a blog post on rmoff.net be if it didn’t include the dirty hack option? 😁 The secret to …
Setting key value when piping from jq to kafkacat
One of my favourite hacks for getting data into Kafka is using kafkacat and stdin, often from jq. You can see this in action with Wi-Fi data, IoT …
Counting the number of messages in a Kafka topic
There’s ways, and then there’s ways, to count the number of records/events/messages in a Kafka topic. Most of them are potentially inaccurate, or …
Why JSON isn’t the same as JSON Schema in Kafka Connect converters and ksqlDB (Viewing Kafka messages bytes as hex)
I’ve been playing around with the new SerDes (serialisers/deserialisers) that shipped with Confluent Platform 5.5 - Protobuf, and JSON Schema (these …
How to install kafkacat on Fedora
kafkacat is one of my go-to tools when working with Kafka. It’s a producer and consumer, but also a swiss-army knife of debugging and troubleshooting …
A quick and dirty way to monitor data arriving on Kafka
I’ve been poking around recently with capturing Wi-Fi packet data and streaming it into Apache Kafka, from where I’m processing and analysing it. …
Streaming Wi-Fi trace data from Raspberry Pi to Apache Kafka with Confluent Cloud
Wi-fi is now ubiquitous in most populated areas, and the way the devices communicate leaves a lot of 'digital exhaust'. Usually a computer …
Primitive Keys in ksqlDB
ksqlDB 0.7 will add support for message keys as primitive data types beyond just STRING (which is all we’ve had to date). That means that Kafka …
Notes on getting data into InfluxDB from Kafka with Kafka Connect
You can download the InfluxDB connector for Kafka Connect here. Documentation for it is here. When a message from your source Kafka topic is written …
Monitoring Sonos with ksqlDB, InfluxDB, and Grafana
I’m quite a fan of Sonos audio equipment but recently had some trouble with some of the devices glitching and even cutting out whilst playing. Under …
Using Kafka Connect and Debezium with Confluent Cloud
This is based on using Confluent Cloud to provide your managed Kafka and Schema Registry. All that you run yourself is the Kafka Connect worker. …
Skipping bad records with the Kafka Connect JDBC sink connector
The Kafka Connect framework provides generic error handling and dead-letter queue capabilities which are available for problems with …
Copying data between Kafka clusters with Kafkacat
kafkacat gives you Kafka super powers 😎 I’ve written before about kafkacat and what a great tool it is for doing lots of useful things as a developer …
Reset Kafka Connect Source Connector Offsets
Kafka Connect in distributed mode uses Kafka itself to persist the offsets of any source connectors. This is a great way to do things as it means …
Manually delete a connector from Kafka Connect
Kafka Connect has as REST API through which all config should be done, including removing connectors that have been created. Sometimes though, you …