Sep 18, 2020
Sep 18, 2020

Using the Debezium MS SQL connector with ksqlDB embedded Kafka Connect

Prompted by a question on StackOverflow I thought I’d take a quick look at setting up ksqlDB to ingest CDC events from Microsoft SQL Server using Debezium. Some of this is based on my previous article, Streaming data from SQL Server to Kafka to Snowflake ❄️ with Kafka Connect. Setting up the Docker Compose I like standalone, repeatable, demo code. For that reason I love using Docker Compose and I embed everything in there - connector installation, the kitchen sink - the works.
Sep 18, 2020
Sep 18, 2020

Including content from external links with Asciidoc in Hugo

I use Hugo for my blog, hosted on GitHub pages. One of the reasons I’m really happy with it is that I can use Asciidoc to author my posts. I was writing a blog recently in which I wanted to include some code that’s hosted on GitHub. I could have copied & pasted it into the blog but that would be lame! With Asciidoc you can use the include:: directive to include both local files:
Sep 11, 2020
Sep 11, 2020

What is Kafka Connect?

Kafka Connect is the integration API for Apache Kafka. Check out this video for an overview of what Kafka Connect enables you to do, and how to do it.

Sep 8, 2020
Sep 8, 2020

Counting the number of messages in a Kafka topic

There’s ways, and then there’s ways, to count the number of records/events/messages in a Kafka topic. Most of them are potentially inaccurate, or inefficient, or both. Here’s one that falls into the potentially inefficient category, using kafkacat to read all the messages and pipe to wc which with the -l will tell you how many lines there are, and since each message is a line, how many messages you have in the Kafka topic:

$ kafkacat -b broker:29092 -t mytestopic -C -e -q| wc -l
       3
Sep 7, 2020
Sep 7, 2020

Poking around the search engines in Google Chrome

Google Chrome automagically adds sites that you visit which support searching to a list of custom search engines. For each one you can set a keyword which activates it, so based on the above list if I want to search Amazon I can just type a <tab> and then my search term

searchengines02
Jul 23, 2020
Jul 23, 2020

Telegram bot - BOT_COMMAND_INVALID

A tiny snippet since I wasted 10 minutes going around the houses on this one…

tl;dr: If you try to create a command that is not in lower case (e.g. Alert not alert) then the setMyCommands API will return BOT_COMMAND_INVALID

Jul 17, 2020
Jul 17, 2020

Learning Golang (some rough notes) - S02E08 - Checking Kafka advertised.listeners with Go

At the beginning of all this my aim was to learn something new (Go), and use it to write a version of a utility that I’d previously hacked together in Python that checks your Apache Kafka broker configuration for possible problems with the infamous advertised.listeners setting. Check out a blog that I wrote which explains all about Apache Kafka and listener configuration. You can find the code at https://github.com/rmoff/kafka-listeners It’s been a fun journey, and now I am pleased to be able to show the results of it.
Jul 16, 2020
Jul 16, 2020

Learning Golang (some rough notes) - S02E07 - Splitting Go code into separate source files and building a binary executable

So far I’ve been running all my code either in the Go Tour sandbox, using Go Playground, or from a single file in VS Code. My explorations in the previous article ended up with a a source file that was starting to get a little bit unwieldily, so let’s take a look at how that can be improved. Within my most recent code, I have the main function and the doProduce function, which is fine when collapsed down: