Common mistakes made when configuring multiple Kafka Connect workers
Kafka Connect can be deployed in two modes: Standalone or Distributed. You can learn more about them in my Kafka Summit London 2019 talk. I usually …
Streaming data from SQL Server to Kafka to Snowflake ❄️ with Kafka Connect
Snowflake is the data warehouse built for the cloud, so let’s get all ☁️ cloudy and stream some data from Kafka running in Confluent Cloud to …
Running Dockerised Kafka Connect worker on GCP
I talk and write about Kafka and Confluent Platform a lot, and more and more of the demos that I’m building are around Confluent Cloud. This means …
Using Kafka Connect and Debezium with Confluent Cloud
This is based on using Confluent Cloud to provide your managed Kafka and Schema Registry. All that you run yourself is the Kafka Connect worker. …
Skipping bad records with the Kafka Connect JDBC sink connector
The Kafka Connect framework provides generic error handling and dead-letter queue capabilities which are available for problems with …
Kafka Connect and Elasticsearch
I use the Elastic stack for a lot of my talks and demos because it complements Kafka brilliantly. A few things have changed in recent releases and …
Reset Kafka Connect Source Connector Offsets
Kafka Connect in distributed mode uses Kafka itself to persist the offsets of any source connectors. This is a great way to do things as it means …
Starting a Kafka Connect sink connector at the end of a topic
When you create a sink connector in Kafka Connect, by default it will start reading from the beginning of the topic and stream all of the …
Resetting a Consumer Group in Kafka
I’ve been using Replicator as a powerful way to copy data from my Kafka rig at home onto my laptop’s Kafka environment. It means that when I’m on the …
Manually delete a connector from Kafka Connect
Kafka Connect has as REST API through which all config should be done, including removing connectors that have been created. Sometimes though, you …
Automatically restarting failed Kafka Connect tasks
Here’s a hacky way to automatically restart Kafka Connect connectors if they fail. Restarting automatically only makes sense if it’s a transient …
Putting Kafka Connect passwords in a separate file / externalising secrets
Kafka Connect configuration is easy - you just write some JSON! But what if you’ve got credentials that you need to pass? Embedding those in a config …
Deleting a Connector in Kafka Connect without the REST API
Kafka Connect exposes a REST interface through which all config and monitoring operations can be done. You can create connectors, delete them, …
When a Kafka Connect converter is not a converter
Kafka Connect is a API within Apache Kafka and its modular nature makes it powerful and flexible. Converters are part of the API but not always fully …
Reading Kafka Connect Offsets via the REST Proxy
When you run Kafka Connect in distributed mode it uses a Kafka topic to store the offset information for each connector. Because it’s just a Kafka …
Kafka Connect Change Log Level and Write Log to File
By default Kafka Connect sends its output to stdout, so you’ll see it on the console, Docker logs, or wherever. Sometimes you might want to route it …
Docker Tips and Tricks with Kafka Connect, ksqlDB, and Kafka
A few years ago a colleague of mine told me about this thing called Docker, and I must admit I dismissed it as a fad…how wrong was I. Docker, and …
Kafka Connect CLI tricks
I do lots of work with Kafka Connect, almost entirely in Distributed mode—even just with 1 node -> makes scaling out much easier when/if needed. …
Kafka Connect and Oracle data types
The Kafka Connect JDBC Connector by default does not cope so well with: NUMBER columns with no defined precision/scale. You may end up with apparent …
Streaming Data from MongoDB into Kafka with Kafka Connect and Debezium
Disclaimer: I am not a MongoDB person. These steps may or may not be appropriate and proper. But they worked for me :) Feel free to post in comments …