Mar 28, 2019

Exploring KSQL Stream-Stream Joins

Introduction

What can you use stream-stream joins for? Can you use them to join between a stream of orders and stream of related shipments to do useful things? What’s not supported in KSQL, where are the cracks?

Mar 25, 2019

Terminate All KSQL Queries

Before you can drop a stream or table that’s populated by a query in KSQL, you have to terminate any queries upon which the object is dependent. Here’s a bit of jq & xargs magic to terminate all queries that are currently running

Mar 19, 2019

Quick Thoughts on Not Making a Crap Slide Deck

This post is the companion to an earlier one that I wrote about conference abstracts. In the same way that the last one was inspired by reviewing a ton of abstracts and noticing a recurring pattern in my suggestions, so this one comes from reviewing a bunch of slide decks for a forthcoming conference. They all look like good talks, but in several cases these great talks are fighting to get out from underneath the deadening weight of slides.

Herewith follows my highly-opinionated, fairly-subjective, and extremely-terse advice and general suggestions for slide decks. You can also find relating ramblings in this recent post too. My friend and colleague Vik Gamov also wrote a good post on this same topic, and linked to a good video that I’d recommend you watch.

Mar 8, 2019

Using httpie with the Kafka REST Proxy

This shows how to use httpie with the Confluent REST Proxy.

Send data

echo '{"records":[{"value":{"foo":"bar"}}]}' | \
  http POST http://localhost:8082/topics/jsontest \
  Content-Type:application/vnd.kafka.json.v2+json Accept:application/vnd.kafka.v2+json

Mar 1, 2019

Preparing a New Talk

I’ve written quite a few talks over the years, but usually as a side-line to my day job. In my role as a Developer Advocate, talks are part of What I Do, and so I can dedicate more time to it. A lot of the talks I’ve done previously have evolved through numerous iterations, and with a new talk to deliver for the "Spring Season" of conferences, I thought it would be interesting to track what it took from concept to actual delivery.

Feb 9, 2019

Travelling for Work, with Kids at Home

I began travelling for my job when my first child was three months old. But don’t mistake correlation for causation…it wasn’t the broken nights' sleep that forced me onto the road, but an excellent job opportunity that seemed worth the risk. Nearly eight years later and I’m in a different job but still with a bunch of travel involved. How much I travel has varied. It’s tended to average around 30%, but has peaked at way more than that.

Jan 29, 2019

Kafka Connect Change Log Level and Write Log to File

By default Kafka Connect sends its output to stdout, so you’ll see it on the console, Docker logs, or wherever. Sometimes you might want to route it to file, and you can do this by reconfiguring log4j. You can also change the configuration to get more (or less) detail in the logs by changing the log level. Finding the log configuration file The configuration file is called connect-log4j.properties and usually found in etc/kafka/connect-log4j.

Jan 21, 2019

Replacing UTF8 non-breaking-space with bash/sed on the Mac

A script I’d batch-run on my Markdown files had inserted a UTF-8 non-breaking-space between Markdown heading indicator and the text, which meant that # My title actually got rendered as that, instead of an H3 title.

Looking at the file contents, I could see it wasn’t just a space between the # and the text, but a non-breaking space.

Jan 21, 2019

How KSQL handles case

KSQL is generally case-sensitive. Very sensitive, at times ;-)

Jan 17, 2019

KSQL REST API cheatsheet

Full reference is here

Jan 17, 2019

Confluent Schema Registry REST API cheatsheet

The Schema Registry support a REST API for finding out information about the schemas within it. Here’s a quick cheatsheat with REST calls that I often use.

Jan 9, 2019

What to Do When Docker on the Mac Runs Out of Space

I use Docker and Docker Compose a lot. Like, every day. It’s a fantastic way to build repeatable demos and examples, that can be torn down and spun up in a repeatable way. But…what happens when the demo that was working is spun up and then tail spins down in a blaze of flames?

Dec 19, 2018

Quick Thoughts on Not Writing a Crap Abstract

I’ve reviewed a bunch of abstracts in the last couple of days, here are some common suggestions I made: No need to include your company name in the abstract text. Chances are I’ve not heard of your company, and even if I have, what does it add to my comprehension of your abstract and what you’re going to talk about? Possible exception would be the "hot" tech companies where people will see a talk just because it’s Netflix etc

Dec 17, 2018

Moving from Ghost to Hugo

Why? I’ve been blogging for quite a few years now, starting on Blogger, soon onto WordPress, and then to Ghost a couple of years ago. Blogger was fairly lame, WP yucky, but I really do like Ghost. It’s simple and powerful and was perfect for my needs. My needs being, an outlet for technical content that respected formatting, worked with a markup language (Markdown), and didn’t f**k things up in the way that WP often would in its WYSIWYG handling of content.

Dec 17, 2018

Pull new version of multiple Docker images

Tiny little snippet this one. Given a list of images:

$ docker images|grep confluent
confluentinc/cp-enterprise-kafka                5.0.0               d0c5528d7f99        3 months ago        600MB
confluentinc/cp-kafka                           5.0.0               373a4e31e02e        3 months ago        558MB
confluentinc/cp-zookeeper                       5.0.0               3cab14034c43        3 months ago        558MB
confluentinc/cp-ksql-server                     5.0.0               691bc3c1991f        4 months ago        493MB
confluentinc/cp-ksql-cli                        5.0.0               e521f3e787d6        4 months ago        488MB
…

Now there’s a new version available, and you want to pull down all the latest ones for it:

docker images|grep "^confluentinc"|awk '{print $1}'|xargs -Ifoo docker pull foo:5.1.0

Dec 15, 2018

Docker Tips and Tricks with KSQL and Kafka

A few years ago a colleague of mine told me about this thing called Docker, and I must admit I dismissed it as a fad…how wrong was I. Docker, and Docker Compose, are one of my key tools of the trade. With them I can build self-contained environments for tutorials, demos, conference talks etc. Tear it down, run it again, without worrying that somewhere a local config changed and will break things.

Dec 12, 2018

Streaming data from Oracle into Kafka (December 2018)

This is a short summary discussing what the options are for integrating Oracle RDBMS into Kafka, as of December 2018. For a more detailed background to why and how at a broader level for all databases (not just Oracle) see this blog and these slides. What techniques & tools are there? Franck Pachot has written up an excellent analysis of the options available here. As of December 2018, this is what the line-up looks like:

Dec 11, 2018

Tools I Use: iPad Pro

I’ve written recently about how I create the diagrams in my blog posts and talks, and from discussions around that, a couple of people were interested more broadly in how I use my iPad Pro. So, on the basis that if two people are interested maybe others are (and if no-one else is, I have a copy-and-paste answer to give to those two people) here we go. Kit iPad Pro 10.

Dec 10, 2018

So how DO you make those cool diagrams?

I write and speak lots about Kafka, and get a fair few questions from this. The most common question is actually nothing to do with Kafka, but instead: How do you make those cool diagrams? So here’s a short, and longer, answer! Update July 2019 I’ve moved away from Paper -> read more here tl;dr An iOS app called Paper, from a company called FiftyThree So, how DO you make those cool diagrams?

Dec 8, 2018

Get mtr working on the Mac

Install Not sure why the brew doesn’t work as it used to, but here’s how to get it working: brew install mtr sudo ln /usr/local/Cellar/mtr/0.92/sbin/mtr /usr/local/bin/mtr sudo ln /usr/local/Cellar/mtr/0.92/sbin/mtr-packet /usr/local/bin/mtr-packet (If you don’t do the two symbolic links (ln) you’ll get mtr: command not found or mtr: Failure to start mtr-packet: Invalid argument) Run sudo mtr google.com