Jun 19, 2020
Jun 19, 2020

How to install connector plugins in Kafka Connect

Kafka Connect (which is part of Apache Kafka) supports pluggable connectors, enabling you to stream data between Kafka and numerous types of system, including to mention just a few:

  • Databases

  • Message Queues

  • Flat files

  • Object stores

The appropriate plugin for the technology which you want to integrate can be found on Confluent Hub.

Jun 17, 2020
Jun 17, 2020

Loading CSV data into Kafka

For whatever reason, CSV still exists as a ubiquitous data interchange format. It doesn’t get much simpler: chuck some plaintext with fields separated by commas into a file and stick .csv on the end. If you’re feeling helpful you can include a header row with field names in.

order_id,customer_id,order_total_usd,make,model,delivery_city,delivery_company,delivery_address
1,535,190899.73,Dodge,Ram Wagon B350,Sheffield,DuBuque LLC,2810 Northland Avenue
2,671,33245.53,Volkswagen,Cabriolet,Edinburgh,Bechtelar-VonRueden,1 Macpherson Crossing

In this article we’ll see how to load this CSV data into Kafka, without even needing to write any code

May 26, 2020
May 26, 2020

Working with JSON nested arrays in ksqlDB - example

Question from the Confluent Community Slack group:

How can I access the data in object in an array like below using ksqlDB stream

"Total": [
        {
          "TotalType": "Standard",
          "TotalAmount": 15.99
        },
{
          "TotalType": "Old Standard",
          "TotalAmount": 16,
" STID":56
        }
]
May 18, 2020
May 18, 2020

Searching Alfred’s Clipboard history programatically

Alfred is one of my favourite productivity apps for the Mac. It’s a file indexer, a clipboard manager, a snippet expander - and that’s just scratching the surface really. I recently got a new machine without it installed and realised just how much I rely on Alfred, particularly its clipboard manager.

May 18, 2020
May 18, 2020

Building a Telegram bot with Apache Kafka and ksqlDB

Imagine you’ve got a stream of data; it’s not “big data,” but it’s certainly a lot. Within the data, you’ve got some bits you’re interested in, and of those bits, you’d like to be able to query information about them at any point. Sounds fun, right? What if you didn’t need any datastore other than Apache Kafka itself to be able to do this? What if you could ingest, filter, enrich, aggregate, and query data with just Kafka?
May 4, 2020
May 4, 2020

Add Markers list from Screenflow to Youtube Table of Contents

Screenflow has a useful Markers feature for adding notes to the timeline. You can use these to helpfully add a table of contents to your Youtube video, but unfortunately Screenflow doesn’t have the option to export them directly. Instead, use the free Subler program as an intermediary (download it from here). Export from Screenflow with a chapters track Open the file in Subler and export to text file From there, tidy up the text file from the source
Apr 20, 2020
Apr 20, 2020

Using Confluent Cloud when there is no Cloud (or internet)

☁️Confluent Cloud is a great solution for a hosted and managed Apache Kafka service, with the additional benefits of Confluent Platform such as ksqlDB and managed Kafka Connect connectors. But as a developer, you won’t always have a reliable internet connection. Train, planes, and automobiles—not to mention crappy hotel or conference Wi-Fi. Wouldn’t it be useful if you could have a replica of your Cloud data on your local machine?
Apr 20, 2020
Apr 20, 2020

How to install kafkacat on Fedora

kafkacat is one of my go-to tools when working with Kafka. It’s a producer and consumer, but also a swiss-army knife of debugging and troubleshooting capabilities. So when I built a new Fedora server recently, I needed to get it installed. Unfortunately there’s no pre-packed install available on yum, so here’s how to do it manually. Note kafkacat is now known as kcat (ref). When invoking the command you will need to use kcat in place of kafkacat.
Apr 16, 2020
Apr 16, 2020

Converting from AsciiDoc to Google Docs and MS Word

Updated 16 April 2020 to cover formatting tricks & add import to Google Docs info Short and sweet this one. I’ve written in the past how I love Markdown but I’ve actually moved on from that and now firmly throw my hat in the AsciiDoc ring. I’ll write another post another time explaining why in more detail, but in short it’s just more powerful whilst still simple and readable without compilation.