May 22, 2019

Deleting a Connector in Kafka Connect without the REST API

Kafka Connect exposes a REST interface through which all config and monitoring operations can be done. You can create connectors, delete them, restart them, check their status, and so on. But, I found a situation recently in which I needed to delete a connector and couldn’t do so with the REST API. Here’s another way to do it, by amending the configuration Kafka topic that Kafka Connect in distributed mode uses to persist configuration information for connectors. Note that this is not a recommended way of working with Kafka Connect—the REST API is there for a good reason :)

Sep 4, 2018

Sending multiline messages to Kafka

(SO answer repost) You can use kafkacat to send messages to Kafka that include line breaks. To do this, use its -D operator to specify a custom message delimiter (in this example /): kafkacat -b kafka:29092 \ -t test_topic_01 \ -D/ \ -P <<EOF this is a string message with a line break/this is another message with two line breaks! EOF Note that the delimiter must be a single byte - multi-byte chars will end up getting included in the resulting message See issue #140

May 10, 2018

Quick ’n Easy Population of Realistic Test Data into Kafka

tl;dr Use curl to pull data from the Mockaroo REST endpoint, and pipe it into kafkacat, thus:

curl -s "https://api.mockaroo.com/api/d5a195e0?count=2&key=ff7856d0"| \
kafkacat -b localhost:9092 -t purchases -P