Dec 14, 2020
Dec 14, 2020

🎄 Twelve Days of SMT 🎄 - Day 5: MaskField

If you want to mask fields of data as you ingest from a source into Kafka, or write to a sink from Kafka with Kafka Connect, the MaskField Single Message Transform is perfect for you. It retains the fields whilst replacing its value.

To use the Single Message Transform you specify the field to mask, and its replacement value. To mask the contents of a field called cc_num you would use:

"transforms"                               : "maskCC",
"transforms.maskCC.type"                   : "org.apache.kafka.connect.transforms.MaskField$Value",
"transforms.maskCC.fields"                 : "cc_num",
"transforms.maskCC.replacement"            : "****-****-****-****"
Dec 11, 2020
Dec 11, 2020

🎄 Twelve Days of SMT 🎄 - Day 4: RegExRouter

If you want to change the topic name to which a source connector writes, or object name that’s created on a target by a sink connector, the RegExRouter is exactly what you need.

To use the Single Message Transform you specify the pattern in the topic name to match, and its replacement. To drop a prefix of test- from a topic you would use:

"transforms"                             : "dropTopicPrefix",
"transforms.dropTopicPrefix.type"        : "org.apache.kafka.connect.transforms.RegexRouter",
"transforms.dropTopicPrefix.regex"       : "test-(.*)",
"transforms.dropTopicPrefix.replacement" : "$1"
Dec 10, 2020
Dec 10, 2020

🎄 Twelve Days of SMT 🎄 - Day 3: Flatten

The Flatten Single Message Transform (SMT) is useful when you need to collapse a nested message down to a flat structure.

To use the Single Message Transform you only need to reference it; there’s no additional configuration required:

"transforms"                    : "flatten",
"transforms.flatten.type"       : "org.apache.kafka.connect.transforms.Flatten$Value"
Dec 9, 2020
Dec 9, 2020

🎄 Twelve Days of SMT 🎄 - Day 2: ValueToKey and ExtractField

Setting the key of a Kafka message is important as it ensures correct logical processing when consumed across multiple partitions, as well as being a requirement when joining to messages in other topics. When using Kafka Connect the connector may already set the key, which is great. If not, you can use these two Single Message Transforms (SMT) to set it as part of the pipeline based on a field in the value part of the message.

To use the ValueToKey Single Message Transform specify the name of the field (id) that you want to copy from the value to the key:

"transforms"                    : "copyIdToKey",
"transforms.copyIdToKey.type"   : "org.apache.kafka.connect.transforms.ValueToKey",
"transforms.copyIdToKey.fields" : "id",
Dec 8, 2020
Dec 8, 2020

🎄 Twelve Days of SMT 🎄 - Day 1: InsertField (timestamp)

You can use the InsertField Single Message Transform (SMT) to add the message timestamp into each message that Kafka Connect sends to a sink.

To use the Single Message Transform specify the name of the field (timestamp.field) that you want to add to hold the message timestamp:

"transforms"                         : "insertTS",
"transforms.insertTS.type"           : "org.apache.kafka.connect.transforms.InsertField$Value",
"transforms.insertTS.timestamp.field": "messageTS"
Dec 3, 2020
Dec 3, 2020

Life as a Developer Advocate, nine months into a pandemic

Back in March 2020 the western world came to somewhat of a juddering halt, thanks to COVID-19. No-one knew then what would happen, but there was the impression that whilst the next few months were a write-off for sure, maybe things would pick up again later in the year. It’s now early December 2020, and nothing is picking up any time soon. Summer provided a respite from the high levels of infection and mortality (in the UK at least), but then numbers spiked again in many places around the world and what was punted down the river back in March is being firmly punted yet again now.
Dec 2, 2020
Dec 2, 2020

My Workstation - 2020

Is a blog even a blog nowadays if it doesn’t include a "Here is my home office setup"?

Thanks to conferences all being online, and thus my talks being delivered from my study—and my habit of posting a #SpeakerSelfie each time I do a conference talk—I often get questions about my setup. Plus, I’m kinda pleased with it so I want to show it off too ;-)

Nov 13, 2020
Nov 13, 2020

Keynote - Why is Replace Fonts greyed out?

Very short & sweet this post, but Google turned up nothing when I was stuck so hopefully I’ll save someone else some head scratching by sharing this.

keynote01
Nov 3, 2020
Nov 3, 2020

Streaming Geopoint data from Kafka to Elasticsearch

Streaming data from Kafka to Elasticsearch is easy with Kafka Connect - you can see how in this tutorial and video.

One of the things that sometimes causes issues though is how to get location data correctly indexed into Elasticsearch as geo_point fields to enable all that lovely location analysis. Unlike data types like dates and numerics, Elasticsearch’s Dynamic Field Mapping won’t automagically pick up geo_point data, and so you have to do two things: