Confluent Schema Registry REST API cheatsheet

The Schema Registry support a REST API for finding out information about the schemas within it. Here’s a quick cheatsheat with REST calls that I often use.

What to Do When Docker on the Mac Runs Out of Space

I use Docker and Docker Compose a lot. Like, every day. It’s a fantastic way to build repeatable demos and examples, that can be torn down and spun up in a repeatable way. But…what happens when the demo that was working is spun up and then tail spins down in a blaze of flames? Here’s the excerpt of a log from my Oracle container: $ docker-compose up -d ; docker-compose logs -f oracle Recreating connect-jdbc_oracle_1_4be0ad4479f8 .

Quick Thoughts on Not Writing a Crap Abstract

I’ve reviewed a bunch of abstracts in the last couple of days, here are some common suggestions I made: No need to include your company name in the abstract text. Chances are I’ve not heard of your company, and even if I have, what does it add to my comprehension of your abstract and what you’re going to talk about? Possible exception would be the "hot" tech companies where people will see a talk just because it’s Netflix etc

Moving from Ghost to Hugo

Why? I’ve been blogging for quite a few years now, starting on Blogger, soon onto WordPress, and then to Ghost a couple of years ago. Blogger was fairly lame, WP yucky, but I really do like Ghost. It’s simple and powerful and was perfect for my needs. My needs being, an outlet for technical content that respected formatting, worked with a markup language (Markdown), and didn’t f**k things up in the way that WP often would in its WYSIWYG handling of content.

Pull new version of multiple Docker images

Tiny little snippet this one. Given a list of images: $ docker images|grep confluent confluentinc/cp-enterprise-kafka 5.0.0 d0c5528d7f99 3 months ago 600MB confluentinc/cp-kafka 5.0.0 373a4e31e02e 3 months ago 558MB confluentinc/cp-zookeeper 5.0.0 3cab14034c43 3 months ago 558MB confluentinc/cp-ksql-server 5.0.0 691bc3c1991f 4 months ago 493MB confluentinc/cp-ksql-cli 5.0.0 e521f3e787d6 4 months ago 488MB … Now there’s a new version available, and you want to pull down all the latest ones for it:

Docker Tips and Tricks with KSQL and Kafka

A few years ago a colleague of mine told me about this thing called Docker, and I must admit I dismissed it as a fad…how wrong was I. Docker, and Docker Compose, are one of my key tools of the trade. With them I can build self-contained environments for tutorials, demos, conference talks etc. Tear it down, run it again, without worrying that somewhere a local config changed and will break things.

Streaming data from Oracle into Kafka (December 2018)

This is a short summary discussing what the options are for integrating Oracle RDBMS into Kafka, as of December 2018. For a more detailed background to why and how at a broader level for all databases (not just Oracle) see this blog and these slides. What techniques & tools are there? As of December 2018, this is what the line-up looks like: Query-based CDC The JDBC Connector for Kafka Connect, polls the database for new or changed data based on an incrementing ID column and/or update timestamp Log-based CDC

Tools I Use: iPad Pro

I’ve written recently about how I create the diagrams in my blog posts and talks, and from discussions around that, a couple of people were interested more broadly in how I use my iPad Pro. So, on the basis that if two people are interested maybe others are (and if no-one else is, I have a copy-and-paste answer to give to those two people) here we go. Kit iPad Pro 10.

So how DO you make those cool diagrams?

I write and speak lots about Kafka, and get a fair few questions from this. The most common question is actually nothing to do with Kafka, but instead: How do you make those cool diagrams? So here’s a short, and longer, answer! tl;dr An iOS app called Paper, from a company called FiftyThree So, how DO you make those cool diagrams? Disclaimer: This is a style that I have copied straight from my esteemed colleagues at Confluent, including Neha Narkhede and Ben Stopford, as well as others including Martin Kleppmann.

Get mtr working on the Mac

Install Not sure why the brew doesn’t work as it used to, but here’s how to get it working: brew install mtr sudo ln /usr/local/Cellar/mtr/0.92/sbin/mtr /usr/local/bin/mtr sudo ln /usr/local/Cellar/mtr/0.92/sbin/mtr-packet /usr/local/bin/mtr-packet (If you don’t do the two symbolic links (ln) you’ll get mtr: command not found or mtr: Failure to start mtr-packet: Invalid argument) Run sudo mtr google.com

Kafka Connect CLI tricks

I do lots of work with Kafka Connect, almost entirely in Distributed mode—even just with 1 node -> makes scaling out much easier when/if needed. Because I’m using Distributed mode, I use the Kafka Connect REST API to configure and manage it. Whilst others might use GUI REST tools like Postman etc, I tend to just use the commandline. Here are some useful snippets that I use all the time.

Logging in as root on Oracle Database Docker image

tl;dr:

docker exec --interactive \
            --tty \
            --user root \
            --workdir / \
            oracle-container-name bash

ERROR: Invalid interpolation format for “command” option in service…

Doing some funky Docker Compose stuff, including:

Flatten CDC records in KSQL

The problem - nested messages in Kafka

Data comes into Kafka in many shapes and sizes. Sometimes it’s from CDC tools, and may be nested like this:

Streaming geopoint data from Kafka to Elasticsearch

Using the Elasticsearch Kafka Connect connector to stream events from a Kafka topic to Elasticsearch.

Exploring JMX with jmxterm

Check out the jmxterm repository
Download jmxterm from https://docs.cyclopsgroup.org/jmxterm

Accessing Kafka Docker containers’ JMX from host

See also docs. To help future Googlers… with the Confluent docker images for Kafka, KSQL, Kafka Connect, etc, if you want to access JMX metrics from within, you just need to pass two environment variables: <x>_JMX_HOSTNAME and <x>_JMX_PORT, prefixed by a component name. <x>_JMX_HOSTNAME - the hostname/IP of the JMX host machine, as accessible from the JMX Client. This is used by the JMX client to connect back into JMX, so must be accessible from the host machine running the JMX client.

Sending multiline messages to Kafka

(SO answer repost) You can use kafkacat to send messages to Kafka that include line breaks. To do this, use its -D operator to specify a custom message delimiter (in this example /): kafkacat -b kafka:29092 \ -t test_topic_01 \ -D/ \ -P <<EOF this is a string message with a line break/this is another message with two line breaks! EOF Note that the delimiter must be a single byte - multi-byte chars will end up getting included in the resulting message See issue #140

Window Timestamps in KSQL / Integration with Elasticsearch

KSQL provides the ability to create windowed aggregations. For example, count the number of messages in a 1 minute window, grouped by a particular column: CREATE TABLE RATINGS_BY_CLUB_STATUS AS \ SELECT CLUB_STATUS, COUNT(*) AS RATING_COUNT \ FROM RATINGS_WITH_CUSTOMER_DATA \ WINDOW TUMBLING (SIZE 1 MINUTES) \ GROUP BY CLUB_STATUS; How KSQL, and Kafka Streams, stores the window timestamp associated with an aggregate, has recently changed. See #1497 for details. Whereas previously the Kafka message timestamp (accessible through the KSQL ROWTIME system column) stored the start of the window for which the aggregate had been calculated, this changed in July 2018 to instead be the timestamp of the latest message to update that aggregate value.

Converting from AsciiDoc to MS Word

Short and sweet this one. I’ve written in the past how I love Markdown but I’ve actually moved on from that and now firmly throw my hat in the AsciiDoc ring. I’ll write another post another time explaining why in more detail, but in short it’s just more powerful whilst still simple and readable without compilation. So anyway, I use AsciiDoc (ADOC) for all my technical (and often non-technical) writing now, and from there usually dump it out to HTML which I can share with people as needed: