So you’ve got a code sample you want to share in a presentation, but whilst it looks beautiful in your text-editor with syntax highlighting, it’s fugly in Keynote? You could screenshot it and paste the image into your slide, but you just know that you’ll want to change that code, and end up re-snapshotting it…what a PITA. Better to have a nicely syntax-highlighted code snippet that you can paste as formatted text into Keynote and amend from there as needed.

In this article I demonstrated how to use KSQL to filter streams of network event data. As well as filtering, KSQL can be used to easily enrich streams. In this article we’ll see how this enriched data can be used to drive analysis in Elasticsearch and Kibana—and how KSQL again came into use for building some stream processing as a result of the discovery made. The data came from my home Ubiquiti router, and took two forms:

Hacky way to keep config files in sync when there’s a new version of some software. Caveat : probably completely wrong, may not pick up config entries added in the new version, etc. But, works for me right here right now ;-) So let’s say we have two folders: confluent-4.1.0 confluent-4.1.1 Same structures, different versions. 4.1.0 was set up with our local config in ./etc, that we want to preserve. We can use diff to easily see what’s changed:

The Kafka Connect JDBC Connector by default does not cope so well with: NUMBER columns with no defined precision/scale. You may end up with apparent junk (bytes) in the output, or just errors. TIMESTAMP WITH LOCAL TIME ZONE. Throws JDBC type -102 not currently supported warning in the log. Read more about NUMBER data type in the Oracle docs. tl;dr : How do I make it work? There are several options:

(preserving this StackOverflow answer for posterity and future Googlers) tl;dr When doing a stream-table join, your table messages must already exist (and must be timestamped) before the stream messages. If you re-emit your source stream messages, after the table topic is populated, the join will succeed. Example data Use kafakcat to populate topics: kafkacat -b localhost:9092 -P -t sessionDetails <<EOF {"Media":"Foo","SessionIdTime":"2018-05-17 11:25:33 BST","SessionIdSeq":1} {"Media":"Foo","SessionIdTime":"2018-05-17 11:26:33 BST","SessionIdSeq":2} EOF kafkacat -b localhost:9092 -P -t voipDetails <<EOF {"SessionIdTime":"2018-05-17 11:25:33 BST","SessionIdSeq":1,"Details":"Bar1a"} {"SessionIdTime":"2018-05-17 11:25:33 BST","SessionIdSeq":1,"Details":"Bar1b"} {"SessionIdTime":"2018-05-17 11:26:33 BST","SessionIdSeq":2,"Details":"Bar2"} EOF Validate topic contents:

tl;dr Use curl to pull data from the Mockaroo REST endpoint, and pipe it into kafkacat, thus:

curl -s "https://api.mockaroo.com/api/d5a195e0?count=2&key=ff7856d0"| \
kafkacat -b localhost:9092 -t purchases -P

So the last post here was in 2011…seven years later I should probably post again, just to point random Google visitors to : My new blog : https://rmoff.net (Wordpress is icky; Ghost FTW!) My employer’s blog on which I also write: http://cnfl.io/rmoff

Disclaimer: I am not a MongoDB person. These steps may or may not be appropriate and proper. But they worked for me :) Feel free to post in comments if I’m doing something wrong MongoDB config - enabling replica sets For Debezium to be able to stream changes from MongoDB, Mongo needs to have replication configured: Docs: Replication / Convert a Standalone to a Replica Set Stop Mongo: rmoff@proxmox01 ~> sudo service mongod stop Add replica set config to /etc/mongod.

DISCLAIMER: I am not a MongoDB person (even if it is Web Scale X-D) - below instructions may work for you, they may not. Use with care! For some work I’ve been doing I wanted to access the data in Ubiquiti’s Unifi controller which it stores in MongoDB. Because I didn’t want to risk my actual Unifi device by changing local settings to enable remote access, and also because the version of MongoDB on it is older than ideal, I wanted to clone the data elsewhere.

Debezium is a CDC tool that can stream changes from MySQL, MongoDB, and PostgreSQL into Kafka, using Kafka Connect. In this article we’ll see how to set it up and examine the format of the data. A subsequent article will show using this realtime stream of data from a RDBMS and join it to data originating from other sources, using KSQL. The software versions used here are: Confluent Platform 4.0 Debezium 0.