May 18, 2020
May 18, 2020

Building a Telegram bot with Apache Kafka and ksqlDB

Imagine you’ve got a stream of data; it’s not “big data,” but it’s certainly a lot. Within the data, you’ve got some bits you’re interested in, and of those bits, you’d like to be able to query information about them at any point. Sounds fun, right? What if you didn’t need any datastore other than Apache Kafka itself to be able to do this? What if you could ingest, filter, enrich, aggregate, and query data with just Kafka?
May 4, 2020
May 4, 2020

Add Markers list from Screenflow to Youtube Table of Contents

Screenflow has a useful Markers feature for adding notes to the timeline. You can use these to helpfully add a table of contents to your Youtube video, but unfortunately Screenflow doesn’t have the option to export them directly. Instead, use the free Subler program as an intermediary (download it from here). Export from Screenflow with a chapters track Open the file in Subler and export to text file From there, tidy up the text file from the source
Apr 20, 2020
Apr 20, 2020

Using Confluent Cloud when there is no Cloud (or internet)

☁️Confluent Cloud is a great solution for a hosted and managed Apache Kafka service, with the additional benefits of Confluent Platform such as ksqlDB and managed Kafka Connect connectors. But as a developer, you won’t always have a reliable internet connection. Train, planes, and automobiles—not to mention crappy hotel or conference Wi-Fi. Wouldn’t it be useful if you could have a replica of your Cloud data on your local machine?
Apr 20, 2020
Apr 20, 2020

How to install kafkacat on Fedora

kafkacat is one of my go-to tools when working with Kafka. It’s a producer and consumer, but also a swiss-army knife of debugging and troubleshooting capabilities. So when I built a new Fedora server recently, I needed to get it installed. Unfortunately there’s no pre-packed install available on yum, so here’s how to do it manually. Note kafkacat is now known as kcat (ref). When invoking the command you will need to use kcat in place of kafkacat.
Apr 16, 2020
Apr 16, 2020

Converting from AsciiDoc to Google Docs and MS Word

Updated 16 April 2020 to cover formatting tricks & add import to Google Docs info Short and sweet this one. I’ve written in the past how I love Markdown but I’ve actually moved on from that and now firmly throw my hat in the AsciiDoc ring. I’ll write another post another time explaining why in more detail, but in short it’s just more powerful whilst still simple and readable without compilation.
Apr 16, 2020
Apr 16, 2020

A quick and dirty way to monitor data arriving on Kafka

I’ve been poking around recently with capturing Wi-Fi packet data and streaming it into Apache Kafka, from where I’m processing and analysing it. Kafka itself is rock-solid - because I’m using ☁️Confluent Cloud and someone else worries about provisioning it, scaling it, and keeping it running for me. But whilst Kafka works just great, my side of the setup—tshark running on a Raspberry Pi—is less than stable. For whatever reason it sometimes stalls and I have to restart the Raspberry Pi and restart the capture process.

Mar 13, 2020
Mar 13, 2020

Are Tech Conferences Dead?

🦠COVID-19 has well and truly hit the tech scene this week. As well as being full of "WFH tips" for all the tech workers suddenly banished from their offices, my particular Twitter bubble is full of DevRel folk musing and debating about what this interruption means to our profession. For sure, in the short term, the Spring conference season is screwed— all the conferences are cancelled (or postponed). But what about the future?
Mar 11, 2020
Mar 11, 2020

Streaming Wi-Fi trace data from Raspberry Pi to Apache Kafka with Confluent Cloud

Wi-fi is now ubiquitous in most populated areas, and the way the devices communicate leaves a lot of 'digital exhaust'. Usually a computer will have a Wi-Fi device that’s configured to connect to a given network, but often these devices can be configured instead to pick up the background Wi-Fi chatter of surrounding devices. There are good reasons—and bad—for doing this. Just like taking apart equipment to understand how it works teaches us things, so being able to dissect and examine protocol traffic lets us learn about this.
Feb 25, 2020
Feb 25, 2020

Kafka Connect JDBC Sink - setting the key field name

I wanted to get some data from a Kafka topic:

ksql> PRINT PERSON_STATS FROM BEGINNING;
Key format: KAFKA (STRING)
Value format: AVRO
rowtime: 2/25/20 1:12:51 PM UTC, key: robin, value: {"PERSON": "robin",
 "LOCATION_CHANGES":1, "UNIQUE_LOCATIONS": 1}

into Postgres, so did the easy thing and used Kafka Connect with the JDBC Sink connector.

Feb 13, 2020
Feb 13, 2020

Adventures in the Cloud, Part 94: ECS

My name’s Robin, and I’m a Developer Advocate. What that means in part is that I build a ton of demos, and Docker Compose is my jam. I love using Docker Compose for the same reasons that many people do:

  • Spin up and tear down fully-functioning multi-component environments with ease. No bespoke builds, no cloning of VMs to preserve "that magic state where everything works"

  • Repeatability. It’s the same each time.

  • Portability. I can point someone at a docker-compose.yml that I’ve written and they can run the same on their machine with the same results almost guaranteed.