Apr 20, 2020

How to install kafkacat on Fedora

kafkacat is one of my go-to tools when working with Kafka. It’s a producer and consumer, but also a swiss-army knife of debugging and troubleshooting capabilities. So when I built a new Fedora server recently, I needed to get it installed. Unfortunately there’s no pre-packed install available on yum, so here’s how to do it manually. Pre-requisite installs We’ll need some packages from the Confluent repo so set this up for yum first by creating /etc/yum.

Continue Reading

Apr 16, 2020

Converting from AsciiDoc to Google Docs and MS Word

Updated 16 April 2020 to cover formatting tricks & add import to Google Docs info Short and sweet this one. I’ve written in the past how I love Markdown but I’ve actually moved on from that and now firmly throw my hat in the AsciiDoc ring. I’ll write another post another time explaining why in more detail, but in short it’s just more powerful whilst still simple and readable without compilation.

Continue Reading

Apr 16, 2020

A quick and dirty way to monitor data arriving on Kafka

I’ve been poking around recently with capturing Wi-Fi packet data and streaming it into Apache Kafka, from where I’m processing and analysing it. Kafka itself is rock-solid - because I’m using ☁️Confluent Cloud and someone else worries about provisioning it, scaling it, and keeping it running for me. But whilst Kafka works just great, my side of the setup—tshark running on a Raspberry Pi—is less than stable. For whatever reason it sometimes stalls and I have to restart the Raspberry Pi and restart the capture process.

Continue Reading

Mar 13, 2020

Are Tech Conferences Dead?

🦠COVID-19 has well and truly hit the tech scene this week. As well as being full of "WFH tips" for all the tech workers suddenly banished from their offices, my particular Twitter bubble is full of DevRel folk musing and debating about what this interruption means to our profession. For sure, in the short term, the Spring conference season is screwed— all the conferences are cancelled (or postponed). But what about the future?

Continue Reading

Mar 11, 2020

Streaming Wi-Fi trace data from Raspberry Pi to Apache Kafka with Confluent Cloud

Wi-fi is now ubiquitous in most populated areas, and the way the devices communicate leaves a lot of 'digital exhaust'. Usually a computer will have a Wi-Fi device that’s configured to connect to a given network, but often these devices can be configured instead to pick up the background Wi-Fi chatter of surrounding devices. There are good reasons—and bad—for doing this. Just like taking apart equipment to understand how it works teaches us things, so being able to dissect and examine protocol traffic lets us learn about this.

Continue Reading

Feb 13, 2020

Adventures in the Cloud, Part 94: ECS

My name’s Robin, and I’m a Developer Advocate. What that means in part is that I build a ton of demos, and Docker Compose is my jam. I love using Docker Compose for the same reasons that many people do:

  • Spin up and tear down fully-functioning multi-component environments with ease. No bespoke builds, no cloning of VMs to preserve "that magic state where everything works"

  • Repeatability. It’s the same each time.

  • Portability. I can point someone at a docker-compose.yml that I’ve written and they can run the same on their machine with the same results almost guaranteed.

Continue Reading

Feb 7, 2020

Primitive Keys in ksqlDB

ksqlDB 0.7 will add support for message keys as primitive data types beyond just STRING (which is all we’ve had to date). That means that Kafka messages are going to be much easier to work with, and require less wrangling to get into the form in which you need them. Take an example of a database table that you’ve ingested into a Kafka topic, and want to join to a stream of events. Previously you’d have had to take the Kafka topic into which the table had been ingested and run a ksqlDB processor to re-key the messages such that ksqlDB could join on them. Friends, I am here to tell you that this is no longer needed!

Continue Reading

Jan 23, 2020

Notes on getting data into InfluxDB from Kafka with Kafka Connect

You can download the InfluxDB connector for Kafka Connect here. Documentation for it is here.

When a message from your source Kafka topic is written to InfluxDB the InfluxDB values are set thus:

  • Timestamp is taken from the Kafka message timestamp (which is either set by your producer, or the time at which it was received by the broker)

  • Tag(s) are taken from the tags field in the message. This field must be a map type - see below

  • Value fields are taken from the rest of the message, and must be numeric or boolean

  • Measurement name can be specified as a field of the message, or hardcoded in the connector config.

Continue Reading

Jan 21, 2020

Monitoring Sonos with ksqlDB, InfluxDB, and Grafana

I’m quite a fan of Sonos audio equipment but recently had some trouble with some of the devices glitching and even cutting out whilst playing. Under the covers Sonos stuff is running Linux (of course) and exposes some diagnostics through a rudimentary frontend that you can access at http://<sonos player IP>:1400/support/review: Whilst this gives you the current state, you can’t get historical data on it. It felt like the problems were happening "

Continue Reading

Jan 16, 2020

Changing the Logging Level for Kafka Connect Dynamically

Logs are magical things. They tell us what an application is doing—or not doing. They help us debug problems. As it happens, they also underpin the entire philosophy of Apache Kafka, but that’s a story for another day. Today we’re talking about logs written by Kafka Connect, and how we can change the amount of detail written. By default, Kafka Connect will write logs at INFO and above. So when it starts up, the settings that it’s using, and any WARN or ERROR messages along the way - a missing configuration, a broken connector, and so on.

Continue Reading

Jan 16, 2020

How to win [or at least not suck] at the conference abstract submission game

Just over a year ago, I put together the crudely-titled "Quick Thoughts on Not Writing a Crap Abstract" after reviewing a few dozen conference abstracts. This time around I’ve had the honour of being on a conference programme committee and with it the pleasure of reading 250+ abstracts—from which I have some more snarky words of wisdom to impart on the matter. Remind me…how does this conference game work? Before we really get into it, let’s recap how this whole game works, because plenty of people are new to conference speaking.

Continue Reading

Jan 9, 2020

Exploring ksqlDB window start time

Prompted by a question on StackOverflow I had a bit of a dig into how windows behave in ksqlDB, specifically with regards to their start time. This article shows also how to create test data in ksqlDB and create data to be handled with a timestamp in the past. For a general background to windowing in ksqlDB see the excellent docs. The nice thing about recent releases of ksqlDB/KSQL is that you can create and populate streams directly with CREATE STREAM and INSERT INTO respectively.

Continue Reading

Jan 8, 2020

Streaming messages from RabbitMQ into Kafka with Kafka Connect

This was prompted by a question on StackOverflow to which I thought the answer would be straightforward, but turned out not to be so. And then I got a bit carried away and ended up with a nice example of how you can handle schema-less data coming from a system such as RabbitMQ and apply a schema to it. Note This same pattern for ingesting bytes and applying a schema will work with other connectors such as JMS, MQTT, and ActiveMQ etc What?

Continue Reading

Dec 20, 2019

Analysing network behaviour with ksqlDB and MongoDB

In this post I want to build on my previous one and show another use of the Syslog data that I’m capturing. Instead of looking for SSH attacks, I’m going to analyse the behaviour of my networking components. Note You can find all the code to run this on GitHub. Getting Syslog data into Kafka As before, let’s create ourselves a syslog connector in ksqlDB:

Continue Reading

Dec 18, 2019

Detecting and Analysing SSH Attacks with ksqlDB

I’ve written previously about ingesting Syslog into Kafka and using KSQL to analyse it. I want to revisit the subject since it’s nearly two years since I wrote about it and some things have changed since then. ksqlDB now includes the ability to define connectors from within it, which makes setting things up loads easier. You can find the full rig to run this on GitHub. Create and configure the Syslog connector To start with, create a source connector:

Continue Reading

Dec 17, 2019

Copy MongoDB collections from remote to local instance

This is revisiting the blog I wrote a while back, which showed using mongodump and mongorestore to copy a MongoDB database from one machine (a Unifi CloudKey) to another. This time instead of a manual lift and shift, I wanted a simple way to automate the update of the target with changes made on the source. The source is as before, Unifi’s CloudKey, which runs MongoDB to store its data about the network - devices, access points, events, and so on.

Continue Reading