Hacky way to keep config files in sync when there’s a new version of some software.

Caveat : probably completely wrong, may not pick up config entries added in the new version, etc. But, works for me right here right now ;-)

So let’s say we have two folders:

confluent-4.1.0
confluent-4.1.1

Same structures, different versions. 4.1.0 was set up with our local config in ./etc, that we want to preserve. We can use diff to easily see what’s changed:

The Kafka Connect JDBC Connector by default does not cope so well with:

• NUMBER columns with no defined precision/scale. You may end up with apparent junk (bytes) in the output, or just errors.
• TIMESTAMP WITH LOCAL TIME ZONE. Throws JDBC type -102 not currently supported warning in the log.

Read more about NUMBER data type in the Oracle docs.

tl;dr : How do I make it work?

There are several options:

New in Confluent Platform 4.1.1 : numeric.mapping

• In the connector configuration, set "numeric.mapping":"best_fit"
• New in Confluent Platform 4.1.1 (Doc)

Avoid the problem in the first place

• Change the DDL of the source object. For example:
  • refine the NUMBER ’s precision and scale
  • Use a TIMESTAMP type that is supported

CAST the datatypes in the query

• Pull from the object directly, and use query in the JDBC connector (instead of table.whitelist)—and cast the columns appropriately:

(preserving this StackOverflow answer for posterity and future Googlers)

tl;dr When doing a stream-table join, your table messages must already exist (and must be timestamped) before the stream messages. If you re-emit your source stream messages, after the table topic is populated, the join will succeed.

Example data

Use kafakcat to populate topics:

kafkacat -b localhost:9092 -P -t sessionDetails <<EOF
{"Media":"Foo","SessionIdTime":"2018-05-17 11:25:33 BST","SessionIdSeq":1}
{"Media":"Foo","SessionIdTime":"2018-05-17 11:26:33 BST","SessionIdSeq":2}
EOF

kafkacat -b localhost:9092 -P -t voipDetails <<EOF
{"SessionIdTime":"2018-05-17 11:25:33 BST","SessionIdSeq":1,"Details":"Bar1a"}
{"SessionIdTime":"2018-05-17 11:25:33 BST","SessionIdSeq":1,"Details":"Bar1b"}
{"SessionIdTime":"2018-05-17 11:26:33 BST","SessionIdSeq":2,"Details":"Bar2"}
EOF

Validate topic contents:

tl;dr Use curl to pull data from the Mockaroo REST endpoint, and pipe it into kafkacat, thus:

curl -s "https://api.mockaroo.com/api/d5a195e0?count=2&key=ff7856d0"| \
kafkacat -b localhost:9092 -t purchases -P

So the last post here was in 2011…seven years later I should probably post again, just to point random Google visitors to :

• My new blog : https://rmoff.net
  • (Wordpress is icky; Ghost FTW!)
• My employer’s blog on which I also write: http://cnfl.io/rmoff

Disclaimer: I am not a MongoDB person. These steps may or may not be appropriate and proper. But they worked for me :) Feel free to post in comments if I’m doing something wrong

MongoDB config - enabling replica sets

For Debezium to be able to stream changes from MongoDB, Mongo needs to have replication configured:

Docs: Replication / Convert a Standalone to a Replica Set

Stop Mongo:

rmoff@proxmox01 ~> sudo service mongod stop

Add replica set config to /etc/mongod.conf:

DISCLAIMER: I am not a MongoDB person (even if it is Web Scale X-D) - below instructions may work for you, they may not. Use with care!

For some work I’ve been doing I wanted to access the data in Ubiquiti’s Unifi controller which it stores in MongoDB. Because I didn’t want to risk my actual Unifi device by changing local settings to enable remote access, and also because the version of MongoDB on it is older than ideal, I wanted to clone the data elsewhere. This article shows you how.

Debezium is a CDC tool that can stream changes from MySQL, MongoDB, and PostgreSQL into Kafka, using Kafka Connect. In this article we’ll see how to set it up and examine the format of the data. A subsequent article will show using this realtime stream of data from a RDBMS and join it to data originating from other sources, using KSQL.

The software versions used here are:

• Confluent Platform 4.0
• Debezium 0.7.2
• MySQL 5.7.19 with Sakila sample database installed

Install Debezium

To use it, you need the relevant JAR for the source system (e.g. MySQL), and make that JAR available to Kafka Connect. Here we’ll set it up for MySQL.

io.confluent.ksql.exception.KafkaTopicException: Topic 'KSQL_NOTIFY' does not conform to the requirements Partitions:1 v 4. Replication: 1 v 1

Why? Because the topic KSQL creates to underpin a CREATE STREAM AS SELECT or CREATE TABLE AS SELECT already exists, and doesn’t match what it expects. By default it will create partitions & replicas based on the same values of the input topic.

Options:

1. Use a different topic, via the WITH (KAFKA_TOPIC='FOO') syntax, e.g.

    CREATE STREAM TEST WITH (KAFKA_TOPIC='FOO') AS SELECT * FROM BAR;
   

2. Tell KSQL to use values that match the existing topic, with the PARTITIONS and REPLICAS parameters. So if the existing topic only has one partition, then tell KSQL that’s what you want:

This article is part of a series exploring Streaming ETL in practice. You can read about setting up the ingest of realtime events from a standard Oracle platform, and building streaming ETL using KSQL.

---

This post shows how we take data streaming in from an Oracle transactional system into Kafka, and simply stream it onwards into Elasticsearch. This is a common pattern, for enabling rapid search or analytics against data held in systems elsewhere.

System:

rmoff@proxmox01:~$ uname -a
Linux proxmox01 4.4.6-1-pve #1 SMP Thu Apr 21 11:25:40 CEST 2016 x86_64 GNU/Linux

rmoff@proxmox01:~$ head -n1 /etc/os-release
PRETTY_NAME="Debian GNU/Linux 8 (jessie)"

rmoff@proxmox01:~$ python --version
Python 2.7.9

Following:

• https://www.confluent.io/blog/introduction-to-apache-kafka-for-python-programmers/
• https://github.com/confluentinc/confluent-kafka-python

Install librdkafka, which is a pre-req for the Python library:

wget -qO - https://packages.confluent.io/deb/4.0/archive.key | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://packages.confluent.io/deb/4.0 stable main"
sudo apt-get install librdkafka-dev python-dev

Setup virtualenv:

sudo apt-get install virtualenv
virtualenv kafka_push_notify
source ./kafka_push_notify/bin/activate.fish

Try to install confluent-kafka:

(This is an expanded version of the intro to an article I posted over on the Confluent blog. Here I get to be as verbose as I like ;))

My first job from university was building a datawarehouse for a retailer in the UK. Back then, it was writing COBOL jobs to load tables in DB2. We waited for all the shops to close and do their end of day system processing, and send their data back to the central mainframe. From there it was checked and loaded, and then reports generated on it. This was nearly twenty years ago as my greying beard will attest—and not a lot has changed in the large majority of reporting and analytics systems since then. COBOL is maybe less common, but what has remained constant is the batch-driven nature of processing. Sometimes batches are run more frequently, and get given fancy names like intra-day ETL or even micro-batching. But batch processing it is, and as such latency is built into our reporting by design. When we opt for batch processing we voluntarily inject delays into the availability of data to our end users. Much better is to build our systems around a streaming platform instead.

This is the detailed step-by-step if you want to recreate the process I describe in the Confluent blog here

---

I used Oracle’s Oracle Developer Days VM, which comes preinstalled with Oracle 12cR2. You can see the notes on how to do this here. These notes take you through installing and configuring:

• Swingbench, to create a sample “Order Entry” schema and simulate events on the Oracle database
• Oracle GoldenGate (OGG, forthwith) and Oracle GoldenGate for Big Data (OGG-BD, forthwith)
  • I’m using Oracle GoldenGate 12.3.1 which includes the Kafka Connect handler as part of its distribution. A connector for earlier versions can be found here. Some of the syntax may differ in the configuration below - if you hit problems then check out an article that I wrote with an earlier version of the tool.
• OGG extract from the Order Entry schema
• Confluent Platform
• KSQL
• Elasticsearch

From this point, I’ll now walk through configuring OGG-BD with the Kafka Connect handler

---

See also Kafka Listeners - Explained

---

A short post to help Googlers. On a single-node sandbox Apache Kafka / Confluent Platform installation, I was getting this error from Schema Registry, Connect, etc:

WARN [AdminClient clientId=adminclient-3] Connection to node -1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

KSQL was throwing a similar error:

KSQL cannot initialize AdminCLient.

I had correctly set the machine’s hostname in my Kafka server.properties:

Some notes that I made on installing and configuring Oracle GoldenGate with Confluent Platform. Excuse the brevity, but hopefully useful to share!

---

I used the Oracle Developer Days VM for this - it’s preinstalled with Oracle 12cR2. Big Data Lite is nice but currently has an older version of GoldenGate.

Login to the VM (oracle/oracle) and then install some useful things:

sudo rpm -Uvh https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
sudo yum install -y screen htop collectl rlwrap p7zip unzip sysstat perf iotop
sudo su -
cd /etc/yum.repos.d/
wget http://download.opensuse.org/repositories/shells:fish:release:2/CentOS_7/shells:fish:release:2.repo
yum install fish

Check Oracle version etc:

Here’s where I’ll be!

If you use Google Calendar you can click on individual entries above and select copy to my calendar - which of course you’ll want to do for all the ones I’ve marked as [SPEAKING] :-)

Here’s a list of all the Apache Kafka talks at OpenWorld and JavaOne, most of which I’ll be trying to get to.

There’s an impressive 19 sessions that cover Apache Kafka™ at Oracle OpenWorld, JavaOne, and Oak Table World this year! You can find the full list with speakers in the session catalogs for OOW, JavaOne, and Oak Table World. OTW is an awesome techie conference which is at the same time as OpenWorld, next door to Moscone. Hope to see you there!

Check out the writeup of my previous visit to OOW including useful tips here.

The Replicat was kapput:

GGSCI (localhost.localdomain) 3> info rkconnoe

REPLICAT   RKCONNOE  Last Started 2017-09-12 17:06   Status ABENDED
Checkpoint Lag       00:00:00 (updated 00:46:34 ago)
Log Read Checkpoint  File /u01/app/ogg/dirdat/oe000000
                     First Record  RBA 0

So checking the OGG error log ggserr.log showed

2017-09-12T17:06:17.572-0400  ERROR   OGG-15051  Oracle GoldenGate Delivery, rkconnoe.prm:  Java or JNI exception:
                              oracle.goldengate.util.GGException: Error detected handling operation added event.
2017-09-12T17:06:17.572-0400  ERROR   OGG-01668  Oracle GoldenGate Delivery, rkconnoe.prm:  PROCESS ABENDING.

So checking the replicat log dirrpt/RKCONNOE_info_log4j.log showed:

Markdown is a plain-text formatting syntax. It enables you write documents in plain text, readable by others in plain text, and optionally rendered into nicely formatted PDF, HTML, DOCX etc.

It’s used widely in software documentation, particularly open-source, because it enables richer formatting than plain-text alone, but without constraining authors or readers to a given software platform.

Platforms such as github natively support Markdown rendering - so you write your README etc in markdown, and when viewed on github it is automagically rendered - without you needing to actually do anything.

I’m excited to be speaking at several conferences and meetups over the next few months. Unsurprisingly, the topic will be Apache Kafka!

If you’re at any of these, please do come and say hi :)

Apache Kafka Meetup - London

My first time talking at the London Apache Kafka Meetup - always a sold-out crowd, this will be fun!

• September 20th, 19:00 : Look Ma, no Code! Building Streaming Data Pipelines with Apache Kafka
  • Slides are available here

Oracle OpenWorld - San Francisco

This will be my second time at OOW - I wrote up my previous trip here