A few years ago a colleague of mine told me about this thing called Docker, and I must admit I dismissed it as a fad…how wrong was I. Docker, and Docker Compose, are one of my key tools of the trade. With them I can build self-contained environments for tutorials, demos, conference talks etc. Tear it down, run it again, without worrying that somewhere a local config changed and will break things.

Continue Reading

This is a short summary discussing what the options are for integrating Oracle RDBMS into Kafka, as of December 2018. For a more detailed background to why and how at a broader level for all databases (not just Oracle) see this blog and these slides. What techniques & tools are there? As of December 2018, this is what the line-up looks like: Query-based CDC The JDBC Connector for Kafka Connect, polls the database for new or changed data based on an incrementing ID column and/or update timestamp Log-based CDC

Continue Reading

I’ve written recently about how I create the diagrams in my blog posts and talks, and from discussions around that, a couple of people were interested more broadly in how I use my iPad Pro. So, on the basis that if two people are interested maybe others are (and if no-one else is, I have a copy-and-paste answer to give to those two people) here we go. ## Kit

Continue Reading

I write and speak lots about Kafka, and get a fair few questions from this. The most common question is actually nothing to do with Kafka, but instead: How do you make those cool diagrams? So here’s a short, and longer, answer! ## tl;dr An iOS app called Paper, from a company called FiftyThree So, how DO you make those cool diagrams? Disclaimer: This is a style that I have copied straight from my esteemed colleagues at Confluent, including Neha Narkhede and Ben Stopford, as well as others including Martin Kleppmann.

Continue Reading

Install Not sure why the brew doesn’t work as it used to, but here’s how to get it working: brew install mtr sudo ln /usr/local/Cellar/mtr/0.92/sbin/mtr /usr/local/bin/mtr sudo ln /usr/local/Cellar/mtr/0.92/sbin/mtr-packet /usr/local/bin/mtr-packet (If you don’t do the two symbolic links (ln) you’ll get mtr: command not found or mtr: Failure to start mtr-packet: Invalid argument) Run sudo mtr google.com

Continue Reading

I do lots of work with Kafka Connect, almost entirely in Distributed mode—even just with 1 node -> makes scaling out much easier when/if needed. Because I’m using Distributed mode, I use the Kafka Connect REST API to configure and manage it. Whilst others might use GUI REST tools like Postman etc, I tend to just use the commandline. Here are some useful snippets that I use all the time.

Continue Reading

tl;dr: docker exec --interactive \ --tty \ --user root \ --workdir / \ oracle-container-name bash Using Oracle’s Docker database image I wanted to install some additional apps, without modifying the Dockerfile. Connect to the container: $ docker exec --interactive --tty docker-compose_oracle_1_479e7fa05ab5 bash No sudo: [oracle@a37d6e99353b ~]$ sudo whoami bash: sudo: command not found Googled, found the the --user flag for Docker, tried that: $ docker exec --interactive --tty --user root docker-compose_oracle_1_479e7fa05ab5 bash OCI runtime exec failed: exec failed: container_linux.

Continue Reading

Doing some funky Docker Compose stuff, including: … kafkacat: image: confluentinc/cp-kafkacat:latest depends_on: - kafka command: - bash - -c - | echo "Waiting for Kafka ⏳" cub kafka-ready -b kafka:29092 1 300 && while [ 1 -eq 1 ] do awk '{print $0;system("sleep 0.5");}' /data/data.json | \ kafkacat -b kafka:29092 -P -t purchases done volumes: - $PWD:/data … Run it: $ docker-compose up -d Oh noes! Error! ERROR: Invalid interpolation format for "command" option in service "kafkacat": "echo 'Waiting for Kafka'

Continue Reading

The problem - nested messages in Kafka Data comes into Kafka in many shapes and sizes. Sometimes it’s from CDC tools, and may be nested like this: { "SCN": 12206116841348, "SEG_OWNER": "KFKUSER", "TABLE_NAME": "CDCTAB2", "TIMESTAMP": 1539162785000, "SQL_REDO": "insert into \"KFKUSER\".\"CDCTAB2\"(\"ID\",\"CITY\",\"NATIONALITY\") values (634789,'AHMEDABAD','INDIA')", "OPERATION": "INSERT", "data": { "value": { "ID": 634789, "CITY": { "string": "AHMEDABAD" }, "NATIONALITY": { "string": "INDIA" } } }, "before": null } Note that the ‘payload’ is nested under data->value.

Continue Reading

Using the Elasticsearch Kafka Connect connector to stream events from a Kafka topic to Elasticsearch. curl -X "POST" "http://kafka-connect:8083/connectors/" \ -H "Content-Type: application/json" \ -d '{ "name": "es_sink_ATM_POSSIBLE_FRAUD", "config": { "topics": "ATM_POSSIBLE_FRAUD", "key.converter": "org.apache.kafka.connect.storage.StringConverter", "value.converter": "org.apache.kafka.connect.json.JsonConverter", "value.converter.schemas.enable": false, "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector", "key.ignore": "true", "schema.ignore": "true", "type.name": "type.name=kafkaconnect", "topic.index.map": "ATM_POSSIBLE_FRAUD:atm_possible_fraud", "connection.url": "http://elasticsearch:9200" } }' Dynamic mapping setup in Elasticsearch (before running the Connector) to force columns to a given type: curl -XPUT "http://elasticsearch:9200/_template/kafkaconnect/" -H 'Content-Type: application/json' -d' { "index_patterns": "*", "settings": { "number_of_shards": 1, "number_of_replicas": 0 }, "mappings": { "_default_": { "dynamic_templates": [ { "dates": { "match": "*TIMESTAMP", "mapping": { "type": "date" } } }, { "geopoint": { "match": "*LOCATION", "mapping": { "type": "geo_point" } } }, { "geopoint2": { "match": "location", "mapping": { "type": "geo_point" } } }, { "non_analysed_string_template": { "match": "account_id, atm, transaction_id", "match_mapping_type": "string", "mapping": { "type": "keyword" } } } ] } } }' Sample JSON message from Kafka: (pretty-printed)

Continue Reading

Check out the jmxterm repository / Download jmxterm from http://wiki.cyclopsgroup.org/jmxterm/ Launch: java -jar ~/Downloads/jmxterm-1.0.0-uber.jar --url localhost:30002 You can pass the jmx host/port directly, or use the open command once jmxterm launches. Once connected, use domains to list available domains $>domains #following domains are available JMImplementation com.sun.management io.confluent.ksql.metrics io.confluent.rest java.lang java.nio java.util.logging kafka.admin.client kafka.consumer kafka.producer kafka.streams [...] Switch to a particular domain: $>domain io.confluent.ksql.metrics #domain is set to io.confluent.ksql.metrics List the available MBeans in a the selected domain (you can also run this without choosing a domain first, to see every MBean, but it’s a long list):

Continue Reading

To help future Googlers… with the Confluent docker images for Kafka, KSQL, Kafka Connect, etc, if you want to access JMX metrics from within, you just need to pass two environment variables: KSQL_JMX_HOSTNAME - the hostname/IP of the host machine. This is used by the JMX client to connect back into JMX, so must be accessible from the host machine running the JMX client. If you’re just running your JMX client locally on the Docker host, you can set this to 127.

Continue Reading

(SO answer repost) You can use kafkacat to send messages to Kafka that include line breaks. To do this, use its -D operator to specify a custom message delimiter (in this example /): kafkacat -b kafka:29092 \ -t test_topic_01 \ -D/ \ -P <<EOF this is a string message with a line break/this is another message with two line breaks! EOF Note that the delimiter must be a single byte - multi-byte chars will end up getting included in the resulting message See issue #140

Continue Reading

KSQL provides the ability to create windowed aggregations. For example, count the number of messages in a 1 minute window, grouped by a particular column: CREATE TABLE RATINGS_BY_CLUB_STATUS AS \ SELECT CLUB_STATUS, COUNT(*) AS RATING_COUNT \ FROM RATINGS_WITH_CUSTOMER_DATA \ WINDOW TUMBLING (SIZE 1 MINUTES) \ GROUP BY CLUB_STATUS; How KSQL, and Kafka Streams, stores the window timestamp associated with an aggregate, has recently changed. See #1497 for details. Whereas previously the Kafka message timestamp (accessible through the KSQL ROWTIME system column) stored the start of the window for which the aggregate had been calculated, this changed in July 2018 to instead be the timestamp of the latest message to update that aggregate value.

Continue Reading

Short and sweet this one. I’ve written in the past how I love Markdown but I’ve actually moved on from that and now firmly throw my hat in the AsciiDoc ring. I’ll write another post another time explaining why in more detail, but in short it’s just more powerful whilst still simple and readable without compilation. So anyway, I use AsciiDoc (ADOC) for all my technical (and often non-technical) writing now, and from there usually dump it out to HTML which I can share with people as needed:

Continue Reading

There’s lots going on in the next few months :-) I’m particularly excited to be speaking at several notable conferences for the first time, including JavaZone, USENIX LISA, and Devoxx. As always, if you’re nearby then hope to see you there, and let me know if you want to meet for a coffee or beer! September 🇪🇸 Madrid, Spain 6th Sept: Madrid Kafka Meetup 🇳🇴 Oslo, Norway 10th Sept: Oslo Kafka Meetup 11th Sept: JavaZone 🇧🇪 Antwerp/Brussels, Belgium 25th Sept: Brussels Kafka Meetup 🇪🇸 Barcelona, Spain 26th Sept: Barcelona Kafka Meetup October 🇬🇧 Leeds, UK 4th Oct: The JVM Thing meetup 🇺🇸 Nashville (TN), USA 31st Oct: LISA18 November 🇩🇪 Münich, Germany 7th Nov: W-JAX 🇧🇪 Antwerp, Belgium 13th Nov: Devoxx Belgium 🇵🇱 Krakow, Poland 26th Nov: CoreDump December 🇬🇧 Liverpool, UK 4th Dec: UKOUG TECH 18 🇩🇪 Frankfurt, Germany 10th Dec: Apache Kafka Meetup 11th Dec: IT Days

Continue Reading

This question comes up on StackOverflow and such places a lot, so here’s something to try and help. tl;dr : You need to set advertised.listeners (or KAFKA_ADVERTISED_LISTENERS if you’re using Docker images) to the external address (host/IP) so that clients can correctly connect to it. Otherwise they’ll try to connect to the internal host address–and if that’s not reachable then problems ensue. In this post I’ll talk about why this is necessary, and then show how to do it, based on a couple of scenarios - Docker, and AWS.

Continue Reading

So you’ve got a code sample you want to share in a presentation, but whilst it looks beautiful in your text-editor with syntax highlighting, it’s fugly in Keynote? You could screenshot it and paste the image into your slide, but you just know that you’ll want to change that code, and end up re-snapshotting it…what a PITA. Better to have a nicely syntax-highlighted code snippet that you can paste as formatted text into Keynote and amend from there as needed.

Continue Reading

In this article I demonstrated how to use KSQL to filter streams of network event data. As well as filtering, KSQL can be used to easily enrich streams. In this article we’ll see how this enriched data can be used to drive analysis in Elasticsearch and Kibana—and how KSQL again came into use for building some stream processing as a result of the discovery made. The data came from my home Ubiquiti router, and took two forms:

Continue Reading

Hacky way to keep config files in sync when there’s a new version of some software. Caveat : probably completely wrong, may not pick up config entries added in the new version, etc. But, works for me right here right now ;-) So let’s say we have two folders: confluent-4.1.0 confluent-4.1.1 Same structures, different versions. 4.1.0 was set up with our local config in ./etc, that we want to preserve.

Continue Reading