Some notes that I made on installing and configuring Oracle GoldenGate with Confluent Platform. Excuse the brevity, but hopefully useful to share! I used the Oracle Developer Days VM for this - it’s preinstalled with Oracle 12cR2. Big Data Lite is nice but currently has an older version of GoldenGate. Login to the VM (oracle/oracle) and then install some useful things: sudo rpm -Uvh https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm sudo yum install -y screen htop collectl rlwrap p7zip unzip sysstat perf iotop sudo su - cd /etc/yum.

Here’s where I’ll be! If you use Google Calendar you can click on individual entries above and select copy to my calendar - which of course you’ll want to do for all the ones I’ve marked as [SPEAKING] :-) Here’s a list of all the Apache Kafka talks at OpenWorld and JavaOne, most of which I’ll be trying to get to.

There’s an impressive 19 sessions that cover Apache Kafka™ at Oracle OpenWorld, JavaOne, and Oak Table World this year! You can find the full list with speakers in the session catalogs for OOW, JavaOne, and Oak Table World. OTW is an awesome techie conference which is at the same time as OpenWorld, next door to Moscone. Hope to see you there! Check out the writeup of my previous visit to OOW including useful tips here.

The Replicat was kapput: GGSCI (localhost.localdomain) 3> info rkconnoe REPLICAT RKCONNOE Last Started 2017-09-12 17:06 Status ABENDED Checkpoint Lag 00:00:00 (updated 00:46:34 ago) Log Read Checkpoint File /u01/app/ogg/dirdat/oe000000 First Record RBA 0 So checking the OGG error log ggserr.log showed 2017-09-12T17:06:17.572-0400 ERROR OGG-15051 Oracle GoldenGate Delivery, rkconnoe.prm: Java or JNI exception: oracle.goldengate.util.GGException: Error detected handling operation added event. 2017-09-12T17:06:17.572-0400 ERROR OGG-01668 Oracle GoldenGate Delivery, rkconnoe.prm: PROCESS ABENDING. So checking the replicat log dirrpt/RKCONNOE_info_log4j.

Markdown is a plain-text formatting syntax. It enables you write documents in plain text, readable by others in plain text, and optionally rendered into nicely formatted PDF, HTML, DOCX etc. It’s used widely in software documentation, particularly open-source, because it enables richer formatting than plain-text alone, but without constraining authors or readers to a given software platform. Platforms such as github natively support Markdown rendering - so you write your README etc in markdown, and when viewed on github it is automagically rendered - without you needing to actually do anything.

I’m excited to be speaking at several conferences and meetups over the next few months. Unsurprisingly, the topic will be Apache Kafka! If you’re at any of these, please do come and say hi :) Apache Kafka Meetup - London My first time talking at the London Apache Kafka Meetup - always a sold-out crowd, this will be fun! September 20th, 19:00 : Look Ma, no Code! Building Streaming Data Pipelines with Apache Kafka Slides are available here Oracle OpenWorld - San Francisco This will be my second time at OOW - I wrote up my previous trip here

An error that I see coming up frequently in the Kafka Connect community (e.g. mailing list, Slack group, StackOverflow) is: JsonDeserializer with schemas.enable requires "schema" and "payload" fields and may not contain additional fields or No fields found using key and value schemas for table: foo-bar You can see an explanation, and solution, for the issue in my StackOverflow answer here: https://stackoverflow.com/a/45940013/350613 If you’re using schemas.enable in the Connector configuration, you must have schema and payload as the root-level elements of your JSON message ( Which is pretty much verbatim what the error says 😁), like this:

Grafana API Reference Export all Grafana data sources to data_sources folder mkdir -p data_sources && curl -s "http://localhost:3000/api/datasources" -u admin:admin|jq -c -M '.[]'|split -l 1 - data_sources/ This exports each data source to a separate JSON file in the data_sources folder. Load data sources back in from folder This submits every file that exists in the data_sources folder to Grafana as a new data source definition. for i in data_sources/*; do \ curl -X "POST" "http://localhost:3000/api/datasources" \ -H "Content-Type: application/json" \ --user admin:admin \ --data-binary @$i done

Usually connecting external disks in Linux is easy. Plug it in, run fdisk -l or lsblk | grep disk to identify the device ID, and then mount it. Unfortunately in this instance, plugging in my Seagate 2TB wasn’t so simple. The server is running Proxmox: # uname -a Linux proxmox01 4.4.6-1-pve #1 SMP Thu Apr 21 11:25:40 CEST 2016 x86_64 GNU/Linux No device showed up on lsblk or fdisk -l. In dmesg I saw:

Kafka’s Connect API is a wondrous way of easily bringing data in and out of Apache Kafka without having to write a line of code. By choosing a Connector from the many available, it’s possible to set up and end-to-end data pipeline with just a few lines of configuration. You can configure this by hand, or you can use the Confluent Control Center, for both management and monitoring: BUT … there are times when not all goes well - perhaps your source has gone offline, or one of your targets has been misconfigured.