Kafka Connect is a API within Apache Kafka and its modular nature makes it powerful and flexible. Converters are part of the API but not always fully understood. I’ve written previously about Kafka Connect converters, and this post is just a hands-on example to show even further what they are—and are not—about.
When you run Kafka Connect in distributed mode it uses a Kafka topic to store the offset information for each connector. Because it’s just a Kafka topic, you can read that information using any consumer.
Prompted by a question on StackOverflow, the requirement is to take a series of events related to a common key and for each key output a series of aggregates derived from a changing value in the events. I’ll use the data from the question, based on ticket statuses. Each ticket can go through various stages, and the requirement was to show, per customer, how many tickets are currently at each stage.
Before you can drop a stream or table that’s populated by a query in KSQL, you have to terminate any queries upon which the object is dependent. Here’s a bit of jq & xargs magic to terminate all queries that are currently running
This post is the companion to an earlier one that I wrote about conference abstracts. In the same way that the last one was inspired by reviewing a ton of abstracts and noticing a recurring pattern in my suggestions, so this one comes from reviewing a bunch of slide decks for a forthcoming conference. They all look like good talks, but in several cases these great talks are fighting to get out from underneath the deadening weight of slides.
Herewith follows my highly-opinionated, fairly-subjective, and extremely-terse advice and general suggestions for slide decks. You can also find relating ramblings in this recent post too. My friend and colleague Vik Gamov also wrote a good post on this same topic, and linked to a good video that I’d recommend you watch.
I’ve written quite a few talks over the years, but usually as a side-line to my day job. In my role as a Developer Advocate, talks are part of What I Do, and so I can dedicate more time to it. A lot of the talks I’ve done previously have evolved through numerous iterations, and with a new talk to deliver for the "Spring Season" of conferences, I thought it would be interesting to track what it took from concept to actual delivery.
I began travelling for my job when my first child was three months old. But don’t mistake correlation for causation…it wasn’t the broken nights' sleep that forced me onto the road, but an excellent job opportunity that seemed worth the risk. Nearly eight years later and I’m in a different job but still with a bunch of travel involved. How much I travel has varied. It’s tended to average around 30%, but has peaked at way more than that.
By default Kafka Connect sends its output to stdout, so you’ll see it on the console, Docker logs, or wherever. Sometimes you might want to route it to file, and you can do this by reconfiguring log4j. You can also change the configuration to get more (or less) detail in the logs by changing the log level.
Finding the log configuration file The configuration file is called connect-log4j.properties and usually found in etc/kafka/connect-log4j.
A script I’d batch-run on my Markdown files had inserted a UTF-8 non-breaking-space between Markdown heading indicator and the text, which meant that # My title actually got rendered as that, instead of an H3 title.
Looking at the file contents, I could see it wasn’t just a space between the # and the text, but a non-breaking space.
I use Docker and Docker Compose a lot. Like, every day. It’s a fantastic way to build repeatable demos and examples, that can be torn down and spun up in a repeatable way. But…what happens when the demo that was working is spun up and then tail spins down in a blaze of flames?
I’ve reviewed a bunch of abstracts in the last couple of days, here are some common suggestions I made:
No need to include your company name in the abstract text. Chances are I’ve not heard of your company, and even if I have, what does it add to my comprehension of your abstract and what you’re going to talk about? Possible exception would be the "hot" tech companies where people will see a talk just because it’s Netflix etc
Why? I’ve been blogging for quite a few years now, starting on Blogger, soon onto WordPress, and then to Ghost a couple of years ago. Blogger was fairly lame, WP yucky, but I really do like Ghost. It’s simple and powerful and was perfect for my needs. My needs being, an outlet for technical content that respected formatting, worked with a markup language (Markdown), and didn’t f**k things up in the way that WP often would in its WYSIWYG handling of content.
A few years ago a colleague of mine told me about this thing called Docker, and I must admit I dismissed it as a fad…how wrong was I. Docker, and Docker Compose, are one of my key tools of the trade. With them I can build self-contained environments for tutorials, demos, conference talks etc. Tear it down, run it again, without worrying that somewhere a local config changed and will break things.