A few years ago a colleague of mine told me about this thing called Docker, and I must admit I dismissed it as a fad…how wrong was I. Docker, and Docker Compose, are one of my key tools of the trade. With them I can build self-contained environments for tutorials, demos, conference talks etc. Tear it down, run it again, without worrying that somewhere a local config changed and will break things.

This is a short summary discussing what the options are for integrating Oracle RDBMS into Kafka, as of December 2018 (refreshed June 2020). For a more detailed background to why and how at a broader level for all databases (not just Oracle) see this blog and this talk.

What techniques & tools are there?

Franck Pachot has written up an excellent analysis of the options available here.

I’ve written recently about how I create the diagrams in my blog posts and talks, and from discussions around that, a couple of people were interested more broadly in how I use my iPad Pro. So, on the basis that if two people are interested maybe others are (and if no-one else is, I have a copy-and-paste answer to give to those two people) here we go.

Kit

• iPad Pro 10.5" (2018)
  • 256GB model
  • Apple Pencil
  • Apple Keyboard
  • iPad wallet/protector
• Matte screen protector

Background

I travel quite a lot for work, so want something with a decent battery life for stuff like:

I write and speak lots about Kafka, and get a fair few questions from this. The most common question is actually nothing to do with Kafka, but instead:

How do you make those cool diagrams?

So here’s a short, and longer, answer!

---

Update July 2019

I’ve moved away from Paper -> read more here

---

---

tl;dr

An iOS app called Paper, from a company called FiftyThree

So, how DO you make those cool diagrams?

Disclaimer: This is a style that I have copied straight from my esteemed colleagues at Confluent, including Neha Narkhede and Ben Stopford, as well as others including Martin Kleppmann.

Install

Not sure why the brew doesn’t work as it used to, but here’s how to get it working:

brew install mtr
sudo ln /usr/local/Cellar/mtr/0.92/sbin/mtr /usr/local/bin/mtr
sudo ln /usr/local/Cellar/mtr/0.92/sbin/mtr-packet /usr/local/bin/mtr-packet

(If you don’t do the two symbolic links (ln) you’ll get mtr: command not found or mtr: Failure to start mtr-packet: Invalid argument)

Run

sudo mtr google.com

I do lots of work with Kafka Connect, almost entirely in Distributed mode—even just with 1 node -> makes scaling out much easier when/if needed. Because I’m using Distributed mode, I use the Kafka Connect REST API to configure and manage it. Whilst others might use GUI REST tools like Postman etc, I tend to just use the commandline. Here are some useful snippets that I use all the time.

I’m showing the commands split with a line continuation character (\) but you can of course run them on a single line. You might also choose to get fancy and set the Connect host and port as environment variables etc, but I leave that as an exercise for the reader :)

Doing some funky Docker Compose stuff, including:

The problem - nested messages in Kafka

Data comes into Kafka in many shapes and sizes. Sometimes it’s from CDC tools, and may be nested like this:

• Check out the jmxterm repository
• Download jmxterm from https://docs.cyclopsgroup.org/jmxterm

See also docs.

To help future Googlers… with the Confluent docker images for Kafka, KSQL, Kafka Connect, etc, if you want to access JMX metrics from within, you just need to pass two environment variables: <x>_JMX_HOSTNAME and <x>_JMX_PORT, prefixed by a component name.

• <x>_JMX_HOSTNAME - the hostname/IP of the JMX host machine, as accessible from the JMX Client.

  This is used by the JMX client to connect back into JMX, so must be accessible from the host machine running the JMX client.

(SO answer repost)

You can use kafkacat to send messages to Kafka that include line breaks. To do this, use its -D operator to specify a custom message delimiter (in this example /):

kafkacat -b kafka:29092 \
        -t test_topic_01 \
        -D/ \
        -P <<EOF
this is a string message 
with a line break/this is 
another message with two 
line breaks!
EOF

Note that the delimiter must be a single byte - multi-byte chars will end up getting included in the resulting message See issue #140

KSQL provides the ability to create windowed aggregations. For example, count the number of messages in a 1 minute window, grouped by a particular column:

CREATE TABLE RATINGS_BY_CLUB_STATUS AS \
SELECT CLUB_STATUS, COUNT(*) AS RATING_COUNT \
FROM RATINGS_WITH_CUSTOMER_DATA \
     WINDOW TUMBLING (SIZE 1 MINUTES) \
GROUP BY CLUB_STATUS;

How KSQL, and Kafka Streams, stores the window timestamp associated with an aggregate, has recently changed. See #1497 for details.

Whereas previously the Kafka message timestamp (accessible through the KSQL ROWTIME system column) stored the start of the window for which the aggregate had been calculated, this changed in July 2018 to instead be the timestamp of the latest message to update that aggregate value. This was in Apache Kafka 2.0 and Confluent Platform 5.0, and back-ported to previous versions.

There’s lots going on in the next few months :-)

I’m particularly excited to be speaking at several notable conferences for the first time, including JavaZone, USENIX LISA, and Devoxx.

As always, if you’re nearby then hope to see you there, and let me know if you want to meet for a coffee or beer!

---

September

🇪🇸 Madrid, Spain

• 6th Sept: Madrid Kafka Meetup

🇳🇴 Oslo, Norway

• 10th Sept: Oslo Kafka Meetup
• 11th Sept: JavaZone

🇧🇪 Antwerp/Brussels, Belgium

• 25th Sept: Brussels Kafka Meetup

🇪🇸 Barcelona, Spain

• 26th Sept: Barcelona Kafka Meetup

---

October

🇬🇧 Leeds, UK

• 4th Oct: The JVM Thing meetup

🇺🇸 Nashville (TN), USA

• 31st Oct: LISA18

---

November

🇩🇪 Münich, Germany

• 7th Nov: W-JAX

🇧🇪 Antwerp, Belgium

• 13th Nov: Devoxx Belgium

🇵🇱 Krakow, Poland

• 26th Nov: CoreDump

---

December

🇬🇧 Liverpool, UK

• 4th Dec: UKOUG TECH 18

🇩🇪 Frankfurt, Germany

• 10th Dec: Apache Kafka Meetup
• 11th Dec: IT Days

(This was cross-posted on the Confluent.io blog)

---

This question comes up on StackOverflow and such places a lot, so here’s something to try and help.

tl;dr : You need to set advertised.listeners (or KAFKA_ADVERTISED_LISTENERS if you’re using Docker images) to the external address (host/IP) so that clients can correctly connect to it. Otherwise they’ll try to connect to the internal host address–and if that’s not reachable then problems ensue.

Put another way, courtesy of Spencer Ruport:

LISTENERS are what interfaces Kafka binds to. ADVERTISED_LISTENERS are how clients can connect.