rmoff's random ramblings

Evaluating Claude's dbt Skills: Building an Eval from Scratch

Fri, 13 Mar 2026 15:34:37 +0000

I wanted to explore the extent to which Claude Code could build a data pipeline using dbt without iterative prompting. What difference did skills, models, and the prompt itself make? I’ve written in a separate post about what I found (yes it’s good; no it’s not going to replace data engineers, yet).

In this post I’m going to show how I ran these tests (with Claude) and analysed the results (using Claude), including a pretty dashboard (created by Claude):

How I do, and don't, use AI on this blog

Thu, 12 Mar 2026 11:16:15 +0000

tl;dr

I use AI heavily on this blog.
I don’t use AI to write any content.

As any followers of my blog will have seen recently, I am a big fan of the productivity—and enjoyment—that AI can bring to one’s work. (In fact, I firmly believe that to opt out of using AI is a somewhat negative step to take in terms of one’s career.)

Here’s how I don’t use AI, and never will:

Claude Code isn't going to replace data engineers (yet)

Wed, 11 Mar 2026 15:16:15 +0000

Ten years late (but hopefully not a dollar short) I recently figured out what all the fuss about dbt is about.

Well that’s cute, Robin, you might be saying. Congratulations for catching up on what data/analytics engineers have been doing for years now. But you see, coding by hand is so 2025. Didn’t you hear? AI is going to replace data engineers.

No it’s not (at least, not yet). In fact, used incorrectly, it’ll do a worse job than you. But used right, it’s a kick-ass tool that any data engineer should be adding to their toolbox today ^*. In this article I’ll show you why.

Claude Code in action with dbt

Wed, 11 Mar 2026 15:15:15 +0000

This is an addendum to the main post about using Claude Code with dbt. It shows an excerpt of a Claude session log so you can see exactly what goes on "under the covers".

For full details of the prompt, commentary, and conclusions, see Claude Code isn’t going to replace data engineers (yet).

Here we can see the steps that Claude Code takes as it figures out for itself anomalies in the data and adapts the dbt model to handle them.

AI will fuck you up if you’re not on board

Fri, 06 Mar 2026 15:16:15 +0000

Yes, you’re right

AI slop is ruining the internet.
Given half a chance AI will delete your inbox or worse (even if you work in Safety and Alignment at Meta):

Nothing humbles you like telling your OpenClaw “confirm before acting” and watching it speedrun deleting your inbox. I couldn’t stop it from my phone. I had to RUN to my Mac mini like I was defusing a bomb. pic.twitter.com/XAxyRwPJ5R
— Summer Yue (@summeryue0) February 23, 2026
Low-effort AI contributions are harming the open-source ecosystem.
LLMs hallucinate

…etc etc, ad infinitum.

But you’re also so, so wrong.

Interesting links - February 2026

Fri, 27 Feb 2026 09:06:08 +0000

Phew, what a month! February may be shorter but that’s not diminished the wealth of truly interesting posts I’ve found to share with you this month.

Claude the Instructor

Fri, 20 Feb 2026 15:21:07 +0000

How do you use your LLM coding agent? Mine is usually Claude the proofreader, Claude the bash monkey, Claude the webdev. All these things are about tasks completed. Read this, write that code, fix that web page.

This week I gave Claude a new job.

Ten years late to the dbt party (DuckDB edition)

Thu, 19 Feb 2026 12:28:13 +0000

Apparently, you can teach an old dog new tricks.

Last year I wrote a blog post about building a data processing pipeline using DuckDB to ingest weather sensor data from the UK’s Environment Agency. The pipeline was based around a set of SQL scripts, and whilst it used important data engineering practices like data modelling, it sidestepped the elephant in the room for code-based pipelines: dbt.

Cosplaying as a webdev with Claude Code in January 2026

Tue, 27 Jan 2026 14:42:42 +0000

In which Claude and [A]I play at being webdevs. For some reflections on the bigger picture of AI as a productivity tool for developers, have a look at the companion post to this one.

I used to speak at a lot of conferences and meetups, and published my talks on a site called noti.st. It’s free to use, but you could pay for bells and whistles including a custom domain, which I duly did: talks.rmoff.net.

Reflections of a Developer on LLMs in January 2026

Tue, 27 Jan 2026 13:40:58 +0000

Funnily enough, Charles Dickens was talking about late 18th century Europe rather than the state of AI and LLMs in 2026, but here goes:

It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of light, it was the season of darkness, it was the spring of hope, it was the winter of despair.

Interacting with Developers on Reddit

Fri, 23 Jan 2026 16:25:37 +0000

LLMs are rapidly changing how we use the internet. Remember just a few years ago when you’d search for something on Google and scroll through the results like some kind of Neanderthal? Heck, you might even click through to page 2 if you were feeling spicy.

These days—and, knowing how this stuff ages, I should perhaps be less broad than "these days" and say just "in January 2026"—Google’s AI Overview at the top of the results has got pretty good for basic stuff, making looking at the actual search results less necessary. That’s if folk even get to Google, when they’ve got an LLM close at hand to answer any and every question that they throw at it (regardless of whether it’s a lazy "how do you spell irony" or somewhat more LLM-appropriate "ELI5 nuclear fusion").

Interesting links - January 2026

Tue, 20 Jan 2026 10:14:50 +0000

This is the twelfth edition of this newsletter in its current form. It’s great to see the audience for it growing, and consistently positive reception when I share it. Nice words always inspire me to carry on with it :D The substack edition (which is exactly the same content but sent out by email), is also picking up views and subscribers.

Alternatives to MinIO for single-node local S3

Wed, 14 Jan 2026 09:42:10 +0000

In late 2025 the company behind MinIO decided to abandon it to pursue other commercial interests. As well as upsetting a bunch of folk, it also put the cat amongst the pigeons of many software demos that relied on MinIO to emulate S3 storage locally, not to mention build pipelines that used it for validating S3 compatibility.

In this blog post I’m going to look at some alternatives to MinIO.

A love letter to Raycast ❤️

Thu, 18 Dec 2025 15:59:31 +0000

"What are the must-have apps to install on my new Mac?"… "Which tool makes you the most productive?"… "Do you still use Alfred?"…

All these questions and more—and the answer to all of them is Raycast!

Interesting links - December 2025

Tue, 16 Dec 2025 10:08:00 +0000

Well it’s that time of year already! Whilst munching on a mince pie, enjoy the final Interesting Links for 2025.

It’s been a busy twelve months for me; this time last year I was signing off from my last company, which went on to be acquired—and last week I found out that my current company (Confluent) is to be acquired by IBM. Despite my reaction against any kind of cheese moving, I figure this is going to be an interesting development and a whole new experience for me :)

Using Graph Analysis with Neo4j to Spot Astroturfing on Reddit

Mon, 01 Dec 2025 11:55:22 +0000

Reddit is one of the longer-standing platforms on the internet, bringing together folk to discuss, rant, grumble, and troll others on all sorts of topics, from Kafka to data engineering to nerding out over really bright torches to grumbling about the state of the country—and a whole lot more.

As a social network it’s a prime candidate for using graph analysis to examine how people interact—and in today’s post, hunt down some sneaky shills ;-)

Interesting links - November 2025

Wed, 26 Nov 2025 09:52:09 +0000

Welcome to the 10th edition of Interesting Links. I’ve got over a hundred links for you this month—all of them, IMHO, interesting :)

(AI) Smells on Medium

Tue, 25 Nov 2025 12:46:34 +0000

As part of compiling the monthly interesting links posts, I go through a ton of RSS feeds, sourced from specific blogs that I follow as well as general aggregators. These aggregators include quality sources like InfoQ, and certain tags on lobste.rs. Here I’ll often find some good articles that I missed in my general travels around the social media feeds in the previous month. I also, so you don’t have to, dive into the AI slop-pit that is Medium and various categories feeds. In amongst the detritus and sewage of LLMs left to ramble unchecked are the occasional proverbial diamonds in the rough, which make the sifting worth the effort.

I thought it might be interesting—and a useful vent to preserve my sanity—to note down some of the “smells” I’ve noticed.

Stumbling into AI: Part 6—I've been thinking about Agents and MCP all wrong

Thu, 20 Nov 2025 03:42:10 +0000

Ever tried to hammer a nail in with a potato?

Nor me, but that’s what I’ve felt like I’ve been attempting to do when trying to really understand agents, as well as to come up with an example agent to build.

As I wrote about previously, citing Simon Willison, an LLM agent runs tools in a loop to achieve a goal. Unlike building ETL/ELT pipelines, these were some new concepts that I was struggling to fit to an even semi-plausible real world example.

That’s because I was thinking about it all wrong.

How we built the demo for the Current NOLA Day 2 keynote using Flink and AI

Thu, 06 Nov 2025 14:20:08 +0000

At Current 2025 in New Orleans this year we built a demo for the Day 2 keynote that would automagically summarise what was happening in the room, as reported by members of the audience. Here’s how we did it!

Tech Radar (Nov 2025) - data blips

Wed, 05 Nov 2025 09:53:26 +0000

The latest Thoughtworks TechRadar is out. Here are some of the more data-related ‘blips’ (as they’re called on the radar) that I noticed.

Interesting links - October 2025

Fri, 31 Oct 2025 13:28:23 +0000

What with Current NOLA 2025 happening this week, and some very last minute preparations for the demo at the keynote on day 2, this month’s links roundup is pushing it right up to the wire :) The demo was pretty cool, and finally I have a good example of how this AI stuff actually fits into a workflow ;) I’ll write it up as a blog post (or two, probably)—stay tuned!

Blog Writing for Developers

Wed, 22 Oct 2025 16:00:00 +0000

A presentation about effective blog writing for developers, covering why to blog, what to write about, and how to structure your content.

Stumbling into AI: Part 5—Agents

Mon, 06 Oct 2025 12:47:17 +0000

A short series of notes for myself as I learn more about the AI ecosystem as of Autumn [Fall] 2025. The driver for all this is understanding more about Apache Flink’s Flink Agents project, and Confluent’s Streaming Agents.

I started off this series—somewhat randomly, with hindsight—looking at Model Context Protocol (MCP). It’s a helper technology to make things easier to use and provide a richer experience. Next I tried to wrap my head around Models—mostly LLMs, but also with an addendum discussing other types of model too. Along the lines of MCP, Retrieval Augmented Generation (RAG) is another helper technology that on its own doesn’t do anything but combined with an LLM gives it added smarts. I took a brief moment in part 4 to try and build a clearer understanding of the difference between ML and AI.

So whilst RAG and MCP combined make for a bunch of nice capabilities beyond models such as LLMs alone, what I’m really circling around here is what we can do when we combine all these things: Agents! But…what is an Agent, both conceptually and in practice? Let’s try and figure it out.

Interesting links - September 2025

Tue, 30 Sep 2025 07:04:31 +0000

Sneaking it in just before the end of the month!

It’s a bumper set of links this month—I started with an original backlog of 125 links to get through. Some fell by the wayside, but plenty of others (78, to be precise) made the cut. With no further ado, let’s get cracking!

Stumbling into AI: Part 4—Terminology Tidy-up (and a little rant)

Tue, 16 Sep 2025 13:38:43 +0000

Having looked at MCP, Models, and RAG, I realised that I’ve been mentally skirting around something that I don’t really understand, so I’m going to expose myself to some ridicule here and try to understand better: what’s the difference between AI and ML? Aren’t they just the same?

Stumbling into AI: Part 3—RAG

Fri, 12 Sep 2025 13:10:34 +0000

A short series of notes for myself as I learn more about the AI ecosystem as of September 2025. The driver for all this is understanding more about Apache Flink’s Flink Agents project, and Confluent’s Streaming Agents.

Having poked around MCP and Models, next up is RAG.

RAG has been one of the buzzwords of the last couple of years, with any vendor worth its salt finding a way to crowbar it into their product. I’d been sufficiently put off it by the hype to steer away from actually understanding what it is. In this blog post, let’s fix that—because if I’ve understood it correctly, it’s a pattern that’s not scary at all.

Stumbling into AI: Part 2—Models

Mon, 08 Sep 2025 08:10:34 +0000

Having poked around MCP and got a broad idea of what it is, I want to next look at Models. What used to be as simple as "I used AI" actually boils down into several discrete areas, particularly when one starts looking at using LLMs beyond writing a rap about Apache Kafka in the style of Monty Python and using it to build agents (like the Flink Agents that prompted this exploration in the first place).

Stumbling into AI: Part 1—MCP

Thu, 04 Sep 2025 09:10:34 +0000

The first thing I want to understand better is MCP.

Interesting links - August 2025

Thu, 21 Aug 2025 16:33:14 +0000

You can find previous editions of Interesting Links here.

Kafka to Iceberg - Exploring the Options

Mon, 18 Aug 2025 13:43:31 +0000

You’ve got data in Apache Kafka.

You want to get that data into Apache Iceberg.

What’s the best way to do it?

Perhaps invariably, the answer is: IT DEPENDS. But fear not: here is a guide to help you navigate your way to choosing the best solution for you 🫵.

Connecting Apache Flink SQL to Confluent Cloud Kafka broker

Tue, 22 Jul 2025 11:27:46 +0000

This is a quick blog post to remind me how to connect Apache Flink to a Kafka topic on Confluent Cloud. You may wonder why you’d want to do this, given that Confluent Cloud for Apache Flink is a much easier way to run Flink SQL. But, for whatever reason, you’re here and you want to understand the necessary incantations to get this connectivity to work.

Interesting links - July 2025

Fri, 18 Jul 2025 14:38:04 +0000

First up, allow me a shameless plug for my blog posts this month:

Keeping your Data Lakehouse in Order: Table Maintenance in Apache Iceberg

Mon, 14 Jul 2025 14:43:04 +0000

Iceberg nicely decouples storage from ingest and query (yay!). When we say "decouples" it’s a fancy way of saying "doesn’t do". Which, in the case of ingest and query, is really powerful. It means that we can store data in an open format, populated by one or more tools, and queried by the same, or other tools. Iceberg gets to be very opinionated and optimised around what it was built for (storing tabular data in a flexible way that can be efficiently queried). This is amazing!

But, what Iceberg doesn’t do is any housekeeping on its data and metadata. This means that getting data in and out of Apache Iceberg isn’t where the story stops.

Writing to Apache Iceberg on S3 using Kafka Connect with Glue catalog

Fri, 04 Jul 2025 15:36:21 +0000

Without wanting to mix my temperature metaphors, Iceberg is the new hawtness, and getting data into it from other places is a common task. I wrote previously about using Flink SQL to do this, and today I’m going to look at doing the same using Kafka Connect.

Kafka Connect can send data to Iceberg from any Kafka topic. The source Kafka topic(s) can be populated by a Kafka Connect source connector (such as Debezium), or a regular application producing directly to it.

Interesting links - June 2025

Fri, 27 Jun 2025 12:42:59 +0000

Writing to Apache Iceberg on S3 using Flink SQL with Glue catalog

Tue, 24 Jun 2025 17:12:50 +0000

In this blog post I’ll show how you can use Flink SQL to write to Iceberg on S3, storing metadata about the Iceberg tables in the AWS Glue Data Catalog. First off, I’ll walk through the dependencies and a simple smoke-test, and then put it into practice using it to write data from a Kafka topic to Iceberg.

Digging into Ducklake

Mon, 02 Jun 2025 14:26:15 +0000

After a week’s holiday ("vacation", for y’all in the US) without a glance at anything work-related, what joy to return and find that the DuckDB folk have been busy, not only with the recent 1.3.0 DuckDB release, but also a brand new project called DuckLake.

Here are my brief notes on DuckLake.

Interesting links - May 2025

Fri, 23 May 2025 10:02:42 +0000

Exploring Joins and Changelogs in Flink SQL

Tue, 20 May 2025 12:30:36 +0000

SQL. Three simple letters. Ess Queue Ell. /ˌɛs kjuː ˈɛl/.

In the data world they bind us together, yet separate us.

As the saying goes, England and America are two countries divided by the same language, and the same goes for the batch and streaming world and some elements of SQL.

🏃🚶 The unofficial Current London 2025 Run/Walk 🏃🚶

Fri, 02 May 2025 14:44:52 +0000

Another year, another Current—another 5k run/walk for anyone who’d like to join!

It's Time We Talked About Time: Exploring Watermarks (And More) In Flink SQL

Fri, 25 Apr 2025 15:26:56 +0000

Whether you’re processing data in batch or as a stream, the concept of time is an important part of accurate processing logic.

Because we process data after it happens, there are a minimum of two different types of time to consider:

When it happened, known as Event Time
When we process it, known as Processing Time (or system time or wall clock time)

Interesting links - April 2025

Tue, 22 Apr 2025 10:02:42 +0000

Confluent Cloud for Apache Flink - Exploring the API

Tue, 25 Mar 2025 15:26:13 +0000

Confluent Cloud for Apache Flink gives you access to run Flink workloads using a serverless platform on Confluent Cloud. After poking around the Confluent Cloud API for configuring connectors I wanted to take a look at the same for Flink.

Using the API is useful particularly if you want to script a deployment, or automate a bulk operation that might be tiresome to do otherwise. It’s also handy if you just prefer living in the CLI :)

Interesting links - March 2025

Mon, 24 Mar 2025 10:01:56 +0000

The problem with publishing February’s interesting links at the beginning of the month and now getting around to publishing March’s at the end is that I have nearly two months' worth of links to share 😅 So with no further ado, let’s crack on.

How to create Carousel posts in LinkedIn…without the bullshit

Fri, 21 Mar 2025 17:42:57 +0000

tl;dr: Upload a PDF document in which each slide of the carousel is one page.

I wanted to post a Carousel post in LinkedIn, but had to wade through a million pages of crap in Google from companies trying to sell shit. Here’s how to do it simply.

Building a data pipeline with DuckDB

Thu, 20 Mar 2025 10:01:56 +0000

In this blog post I’m going to explore how as a data engineer in the field today I might go about putting together a rudimentary data pipeline. I’ll take some operational data, and wrangle it into a form that makes it easily pliable for analytics work.

After a somewhat fevered and nightmarish period during which people walked around declaring "Schema on Read" was the future, that "Data is the new oil", and "Look at the size of my big data", the path that is history in IT is somewhat coming back on itself to a more sensible approach to things.

As they say:

What’s old is new

This is good news for me, because I am old and what I knew then is 'new' now ;)

Exporting Notebooks from DuckDB UI

Wed, 19 Mar 2025 17:01:56 +0000

DuckDB added a very cool UI last week and I’ve been using it as my primary interface to DuckDB since.

One thing that bothered me was that the SQL I was writing in the notebooks wasn’t exportable. Since DuckDB uses DuckDB in the background for storing notebooks, getting the SQL out is easy enough.

Kicking the tyres on the new DuckDB UI

Fri, 14 Mar 2025 12:26:16 +0000

I wrote a couple of weeks ago about using DuckDB and Rill Data to explore a new data source that I’m working with. I wanted to understand the data’s structure and distribution of values, as well as how different entities related. This week DuckDB 1.2.1 was released and that little 0.0.1 version boost brought with it the DuckDB UI.

Here I’ll go through the same process as I did before, and see how much of what I was doing can be done in DuckDB alone now.

Creating an HTTP Source connector on Confluent Cloud from the CLI

Thu, 13 Mar 2025 11:29:40 +0000

In this blog article I’ll show you how you can use the confluent CLI to set up a Kafka cluster on Confluent Cloud, the necessary API keys, and then a managed connector. The connector I’m setting up is the HTTP Source (v2) connector.

It’s part of a pipeline that I’m working on to pull in a feed of data from the UK Environment Agency for processing. The data is spread across three endpoints, and one of the nice features of the HTTP Source (v2) connector is that one connector can pull data from more than one endpoint.

Why is kcat showing the wrong topics?

Thu, 13 Mar 2025 10:42:11 +0000

Much as I love kcat (🤫 it’ll always be kafkacat to me…), this morning I nearly fell out with it 👇

😖 I thought I was going stir crazy, after listing topics on a broker and seeing topics from a different broker.

😵 WTF 😵

Write more blog articles, not fewer (Don't leave the scraps on the cutting floor)

Tue, 11 Mar 2025 10:41:44 +0000

Some would say that the perfect blog article takes the reader on a journey on in which the development process looks like this:

Data Wrangling with Flink SQL

Mon, 10 Mar 2025 16:57:44 +0000

The UK Government publishes a lot of its data as open feeds. One that I keep coming back to is the Environment Agency’s flood-monitoring API that gives access to an estate of sensors that provide information about data such as river levels and rainfall.

The data is well-structured and provided across three primary API endpoints. In this blog article I’m going to show you how I use Flink SQL to explore and wrangle these into the kind of form from which I am then going to build a streaming pipeline using them.

Joining two streams of data with Flink SQL

Thu, 06 Mar 2025 15:45:41 +0000

There was a useful question on the Apache Flink Slack recently about joining data in Flink SQL:

How can I join two streams of data by id in Flink, to get a combined view of the latest data?

How to explode nested arrays with Flink SQL

Mon, 03 Mar 2025 14:41:44 +0000

Let’s imagine we’ve got a source of data with a nested array of multiple values. The data is from an IoT device. Each device has multiple sensors, each sensor provides a reading.

Exploring UK Environment Agency data in DuckDB and Rill

Fri, 28 Feb 2025 10:02:33 +0000

The UK Environment Agency publishes a feed of data relating to rainfall and river levels. As a prelude to building a streaming pipeline with this data, I wanted to understand the model of it first.

DuckDB tricks - renaming fields in a SELECT * across tables

Thu, 27 Feb 2025 09:43:20 +0000

I was exploring some new data, joining across multiple tables, and doing a simple SELECT * as I’d not worked out yet which columns I actually wanted. The issue was, the same field name existing in more than one table. This meant that in the results from the query, it wasn’t clear which field came from which table:

Interesting links - February 2025

Mon, 03 Feb 2025 20:49:01 +0000

Here’s a bunch of interesting links and articles about data that I’ve come across recently.

Checkpoint Chronicle - December 2024

Thu, 19 Dec 2024 00:00:00 +0000

This post originally appeared on the Decodable blog.

Welcome to the Checkpoint Chronicle, a monthly roundup of interesting stuff in the data and streaming space. Your hosts and esteemed curators of said content are Gunnar Morling , Robin Moffatt (your editor-in-chief for this edition), and Hans-Peter Grahsl . Feel free to send our way any choice nuggets that you think we should feature in future editions.

Disabling Vale Linting Selectively in Asciidoc

Wed, 11 Dec 2024 12:05:07 +0000

I’m a HUGE fan of Docs as Code in general, and specifically tools like Vale that lint your prose for adherence to style rule.

One thing that had been bugging me though was how to selectively disable Vale for particular sections of a document. Usually linting issues should be addressed at root: either fix the prose, or update the style rule. Either it’s a rule, or it’s not, right?

Sometimes though I’ve found a need to make a particular exception to a rule, or simply needed to skip linting for a particular file. I was struggling with how to do this in Asciidoc. Despite the documentation showing how to, I could never get it to work reliably. Now I’ve taken some time to dig into it, I think I’ve finally understood :)

Exploring Flink CDC

Wed, 11 Dec 2024 00:00:00 +0000

This post originally appeared on the Decodable blog.

Flink CDC is an interesting part of Apache Flink that I’ve been meaning to take a proper look at for some time now. Originally created by Ververica in 2021 and called “CDC Connectors for Apache Flink”, it was donated to live under the Apache Flink project in April 2024.

Streaming Data from Postgres to Snowflake with CDC and Decodable

Tue, 19 Nov 2024 00:00:00 +0000

This post originally appeared on the Decodable blog.

In my last blog post I looked at why you might need CDC. In this post I’m going to put it into practice with probably the most common use case—extracting data from an operational transactional database to store somewhere else for analytics. I’m going to show Postgres to Snowflake, but the pattern is the same for pretty much any combination, such as MySQL to BigQuery, SQL Server to Redshift, and so on.

Checkpoint Chronicle - October 2024

Wed, 30 Oct 2024 00:00:00 +0000

This post originally appeared on the Decodable blog.

Why Do I Need CDC?

Tue, 15 Oct 2024 00:00:00 +0000

This post originally appeared on the Decodable blog.

Whether it’s Understanding CDC or CDC Explained or even Five things about CDC; number four will shock you!, the internet is awash with articles about Change Data Capture (CDC) . CDC is the process of incrementally extracting data change events as they occur within a database. In this post I’d like to take a step back from these and look at the reasons why you might even want to consider CDC in the first place. From here we’ll then build on the requirements we identify to come up with a proposed solution.

Checkpoint Chronicle - September 2024

Thu, 26 Sep 2024 00:00:00 +0000

This post originally appeared on the Decodable blog.

Current 2024 Recap

Thu, 19 Sep 2024 00:00:00 +0000

This post originally appeared on the Decodable blog.

Launched in 2022, the Current conference hosted by Confluent has established itself as one of the leading conferences in the data streaming space. Stemming from Kafka Summit originally, it’s broadened to reflect Confluent’s product portfolio including most notably Apache Flink.

Current 2024 - 5k Fun Run (or Walk)

Mon, 02 Sep 2024 15:11:42 +0000

At Current 24 a few of us will be going for an early run (or walk) on Tuesday morning. Everyone is very welcome!

Adventures with Apache Flink and Delta Lake

Tue, 27 Aug 2024 00:00:00 +0000

This post originally appeared on the Decodable blog.

Delta Lake (or Delta, as it’s often shortened to) is an open-source project from the Linux Foundation that’s primarily backed by Databricks. It’s an open table format (OTF) similar in concept to Apache Iceberg and Apache Hudi. Having previously dug into using Iceberg with both Apache Flink and Decodable , I wanted to see what it was like to use Delta with Flink—and specifically, Flink SQL.

Declarative Resource Management for Real-time ETL with Decodable

Wed, 14 Aug 2024 00:00:00 +0000

This post originally appeared on the Decodable blog.

So you’ve built your first real-time ETL pipeline with Decodable: congratulations! Now what?

Troubleshooting Flink SQL S3 problems

Tue, 06 Aug 2024 00:00:00 +0000

This post originally appeared on the Decodable blog.

You’d think once was enough. Having already written about the trouble that I had getting Flink SQL to write to S3 (including on MinIO) this should now be a moot issue for me. Right? RIGHT?!

How to Migrate from Amazon MSF

Tue, 30 Jul 2024 00:00:00 +0000

This post originally appeared on the Decodable blog.

Amazon Managed Service for Apache Flink (MSF) is one of several providers of hosted Flink. As my colleague Gunnar Morling described in his recent article , it can be used to run a Flink job that you’ve written in Java or Python (PyFlink). But did you know that this isn’t the only way—or perhaps even the best way—to have your Flink jobs run for you?

Sending Data to Apache Iceberg from Apache Kafka with Apache Flink

Thu, 18 Jul 2024 00:00:00 +0000

This post originally appeared on the Decodable blog.

Sometimes it’s not possible to have too much of a good thing, and whilst this blog may look at first-glance rather similar to the one that I published just recently , today we’re looking at a 100% pure Apache solution. Because who knows, maybe you prefer rolling your own tech stacks instead of letting Decodable do it for you 😉.

Decodable vs. Amazon MSF: Getting Started with Flink SQL

Tue, 02 Jul 2024 00:00:00 +0000

This post originally appeared on the Decodable blog.

One of the things that I love about SQL is the power that it gives you to work with data in a declarative manner. I want this thing…go do it. How should it do it? Well that’s the problem for the particular engine, not me. As a language with a pedigree of multiple decades and no sign of waning (despite a wobbly patch for some whilst NoSQL figured out they actually wanted to be NewSQL 😉), it’s the lingua franca of data systems.

How to get data from Apache Kafka to Apache Iceberg on S3 with Decodable

Tue, 18 Jun 2024 00:00:00 +0000

This post originally appeared on the Decodable blog.

Apache Iceberg is an open table format. It combines the benefits of data lakes (open standards, cheap object storage) with the good things that data warehouses have, like first-class support for tables and SQL capabilities including updates to data in place, time-travel, and transactions. With the recent acquisition by Databricks of Tabular—one of the main companies that contribute to Iceberg—it’s clear that Iceberg is winning out as one of the primary contenders in this space.

Checkpoint Chronicle - May 2024

Tue, 28 May 2024 00:00:00 +0000

This post originally appeared on the Decodable blog.

Welcome to the Checkpoint Chronicle, a monthly roundup of interesting stuff in the data and streaming space. Your hosts and esteemed curators of said content are Gunnar Morling and Robin Moffatt (your editor-in-chief for this edition). Feel free to send our way any choice nuggets that you think we should feature in future editions.

How I Try To Keep Up With The Data Tech World (A List of Data Blogs)

Wed, 22 May 2024 13:19:10 +0000

I do my best to try and keep, if not abreast of, then at least aware of what’s going on in the world of data. That includes RDBMS, Event streaming, stream processing, open source data projects, data engineering, object storage, and more. If you’re interested in the same, then you might find this blog useful, because I’m sharing my sources :)

ngrok DNS headaches

Fri, 03 May 2024 10:56:30 +0000

Let’s not bury the lede: it was DNS. However, unlike the meme ("It’s not DNS, it’s never DNS. It was DNS"), I didn’t even have an inkling that DNS might be the problem.

I’m writing a new blog about streaming Apache Kafka data to Apache Iceberg and wanted to provision a local Kafka cluster to pull data from remotely. I got this working nicely just last year using ngrok to expose the broker to the interwebz, so figured I’d use this again. Simple, right?

Nope.

How to stop AWS CLI clearing the screen

Fri, 26 Apr 2024 12:49:50 +0000

After a break from using AWS I had reason to reacquaint myself with it again today, and did so via the CLI. The AWS CLI is pretty intuitive and has a good helptext system, but one thing that kept frustrasting me was that after closing the help text, the screen cleared—so I couldn’t copy the syntax out to use in my command!

The same thing happened when I ran a command that returned output - the screen cleared.

Here’s how to fix either, or both, of these

Flink SQL—Misconfiguration, Misunderstanding, and Mishaps

Wed, 17 Apr 2024 00:00:00 +0000

This post originally appeared on the Decodable blog.

I never meant to write this blog. I had a whole blog series about Flink SQL lined up…and then I started to write it and realised rapidly that one’s initial exposure to Flink and Flink SQL can be somewhat, shall we say, interesting. Interesting, as in the curse, "may you live in interesting times". Because as wonderful and as powerful Flink is, it is not a simple beast to run for yourself, even as a humble developer just trying to try out some SQL.

Checkpoint Chronicle - March 2024

Fri, 22 Mar 2024 00:00:00 +0000

This post originally appeared on the Decodable blog.

Welcome to the Checkpoint Chronicle, a monthly roundup of interesting stuff in the data and streaming space. Your hosts and esteemed curators of said content are Gunnar Morling and Robin Moffatt (your editor-in-chief for this edition). Feel free to send our way any choice nuggets that you think we should feature in future editions.

🏃🚶 The unofficial Kafka Summit London 2024 Run/Walk 🏃🚶

Fri, 15 Mar 2024 15:00:05 +0000

At this year’s Kafka Summit I’m planning to continue the tradition of going for a run (or walk) with anyone who’d like to join in. This started back at Kafka Summit San Francisco in 2019 over the Golden Gate Bridge and has continued since then. Whilst London’s Docklands might not offer quite the same experience it’ll be fun nonetheless.

Apache Flink talks at Kafka Summit London 2024

Fri, 15 Mar 2024 13:54:39 +0000

This year Kafka Summit London includes a dedicated track for talks about Apache Flink. This reflects the continued rise of interest and use of Apache Flink in the streaming community, as well as the focus that Confluent (the hosts of Kafka Summit) has on it.

I’m looking forward to being back at Kafka Summit. I will be speaking on Tuesday afternoon, room hosting on Wednesday morning, and hanging out at the Decodable booth in between too.

Here’s a list of all the Flink talks, including the talk, time, and speaker. You find find more details, and the full Kafka Summit agenda, here.

Exploring the Flink SQL Gateway REST API

Tue, 12 Mar 2024 00:00:00 +0000

This post originally appeared on the Decodable blog.

The SQL Gateway in Apache Flink provides a way to run SQL in Flink from places other than the SQL Client. This includes using a JDBC Driver (which opens up a multitude of clients), a Hive client via the HiveServer2 endpoint , and directly against the REST Endpoint .

Flink SQL and the Joy of JARs

Tue, 27 Feb 2024 00:00:00 +0000

This post originally appeared on the Decodable blog.

I will wager you half of my lottery winnings from 2023[1] that you’re going to encounter this lovely little error at some point on your Flink SQL journey:

Checkpoint Chronicle - February 2024

Thu, 22 Feb 2024 00:00:00 +0000

This post originally appeared on the Decodable blog.

Welcome to the Checkpoint Chronicle, a monthly roundup of interesting stuff in the data and streaming space. Your hosts and esteemed curators of said content are Gunnar Morling and Robin Moffatt (your editor-in-chief for this edition). Feel free to send our way any choice nuggets that you think we should feature in future editions.

Catalogs in Flink SQL—Hands On

Mon, 19 Feb 2024 00:00:00 +0000

This post originally appeared on the Decodable blog.

In the previous blog post I looked at the role of catalogs in Flink SQL, the different types, and some of the quirks around their configuration and use. If you are new to Flink SQL and catalogs, I would recommend reading that post just to make sure you’re not making some of the same assumptions that I mistakenly did when looking at this for the first time.

Catalogs in Flink SQL—A Primer

Fri, 16 Feb 2024 00:00:00 +0000

This post originally appeared on the Decodable blog.

When you’re using Flink SQL you’ll run queries that interact with objects. An INSERT against a TABLE, a SELECT against a VIEW, for example. These objects are defined using DDL—but where do these definitions live?

Antora Deployment to Cloudflare Across Private Repositories with GitHub Actions

Wed, 17 Jan 2024 12:09:23 +0000

At Decodable we migrated our docs platform onto Antora. I wrote previously about my escapades in getting cross-repository authentication working using Private Access Tokens (PAT). These are fine for just a single user, but they’re tied to that user, which isn’t a good practice for deployment in this case.

In this article I’ll show how to use GitHub Apps and Installation Access Tokens (IAT) instead, and go into some detail on how we’ve deployed Antora. Our GitHub repositories are private which makes it extra-gnarly.

Hosting on GitHub Pages? Watch out for Subdomain Hijacking

Tue, 16 Jan 2024 11:50:36 +0000

A friend messaged me late last night with the scary news that Google had emailed him about a ton of spammy subdomains on his own domain.

Any idea how this could have happened, he asked?

1️⃣🐝🏎️🦆 (1BRC in SQL with DuckDB)

Wed, 03 Jan 2024 12:12:32 +0000

Why should the Java folk have all the fun?!

My friend and colleague Gunnar Morling launched a fun challenge this week: how fast can you aggregate and summarise a billion rows of data? Cunningly named The One Billion Row Challenge (1BRC for short), it’s aimed at Java coders to look at new features in the language and optimisation techniques.

Not being a Java coder myself, and seeing how the challenge has already unofficially spread to other communities including Rust and Python I thought I’d join in the fun using what I know best: SQL.

Deploying Antora with GitHub Actions and a private GitHub repo

Tue, 19 Dec 2023 13:35:19 +0000

Antora is a modern documentation site generator with many nice features including sourcing documentation content from one or more separate git repositories. This means that your docs can be kept under source control (yay 🎉) and in sync with the code of the product that they are documenting (double yay 🎉🎉).

As you would expect for a documentation tool, the Antora documentation is thorough but there was one sharp edge involving GitHub that caught me out which I’ll detail here.

Checkpoint Chronicle - December 2023

Wed, 13 Dec 2023 00:00:00 +0000

This post originally appeared on the Decodable blog.

Welcome to the Checkpoint Chronicle, a monthly roundup of interesting stuff in the data and streaming space. Your hosts and esteemed curators of said content are Gunnar Morling and Robin Moffatt —feel free to send our way any choice nuggets that you think we should feature in future editions.

Productivity tools: AI Image Generators

Thu, 07 Dec 2023 19:59:41 +0000

AI, what a load of hyped-up bollocks, right? Yet here I am, legit writing a blog about it and not for the clickbait but…gasp…because it’s actually useful.

Used correctly, it’s just like any other tool on your desktop. It helps you get stuff done quicker, better—or both.

Hugo not detecting changed pages on Mac

Thu, 16 Nov 2023 15:27:22 +0000

I’ve used Hugo for my blog for several years now, and it’s great. One of the things I love about it is the fast build time coupled with it’s live-reload feature. Using this I can edit my source (Markdown or Asciidoc) in one window, hit save, and see the preview update in my browser window next to it pretty much instantaneously. For copy-editing, experimenting with images, etc this is really helpful.

Learning Apache Flink S01E06: The Flink JDBC Driver

Thu, 16 Nov 2023 15:20:20 +0000

As a newcomer to Apache Flink one of the first things I did was join the Slack community (which is vendor-neutral and controlled by the Flink PMC). At the moment I’m pretty much in full-time lurker mode, soaking up the kind of questions that people have and how they’re using Flink.

One question that caught my eye was from Marco Villalobos, in which he asked about the Flink JDBC driver and a SQLDataException he was getting with a particular datatype. Now, unfortunately, I have no idea about the answer to this question—but the idea of a JDBC driver through which Flink SQL could be run sounded like a fascinating path to follow after previously looking at the SQL Client.

Checkpoint Chronicle - November 2023

Tue, 14 Nov 2023 00:00:00 +0000

This post originally appeared on the Decodable blog.

Welcome to the Checkpoint Chronicle, a monthly roundup of interesting stuff in the data and streaming space. Your hosts and esteemed curators of said content are Gunnar Morling and Robin Moffatt - feel free to send our way any choice nuggets that you think we should feature in future editions.

Using Apache Kafka with ngrok

Wed, 01 Nov 2023 10:07:58 +0000

Sometimes you might want to access Apache Kafka that’s running on your local machine from another device not on the same network. I’m not sure I can think of a production use-case, but there are a dozen examples for sandbox, demo, and playground environments.

In this post we’ll see how you can use ngrok to, in their words, Put localhost on the internet. And specifically, your local Kafka broker on the internet.

Learning Apache Flink S01E05: Installing PyFlink (with some bumps along the way…)

Wed, 25 Oct 2023 15:27:22 +0000

When I started my journey learning Apache Flink one of the things that several people expressed an interest in hearing more about was PyFlink. This appeals to me too, because whilst Java is just something I don’t know and feels beyond me to try and learn, Python is something that I know enough of to at least hack my way around it. I’ve previously had fun with PySpark, and whilst Flink SQL will probably be one of my main focusses, I also want to get a feel for PyFlink.

The first step to using PyFlink is installing it - which should be simple, right?

Learning Apache Flink S01E04: A [Partial] Exploration of the Flink SQL Client

Tue, 10 Oct 2023 16:27:22 +0000

So far I’ve plotted out a bit of a map for my exploration of Apache Flink, looked at what Flink is, and run my first Flink application. Being an absolutely abysmal coder—but knowing a thing or two about SQL—I figure that Flink SQL is where my focus is going to lie (I’m also intrigued by PyFlink, but that’s for another day…).

Learning Apache Flink S01E03: Running my First Flink Cluster and Application

Thu, 05 Oct 2023 14:29:02 +0000

🎉 I just ran my first Apache Flink cluster and application on it 🎉

cd: string not in pwd

Wed, 04 Oct 2023 15:36:35 +0000

A brief diversion from my journey learning Apache Flink to document an interesting zsh oddity that briefly tripped me up:

cd: string not in pwd: flink-1.17.1

Learning Apache Flink S01E02: What is Flink?

Mon, 02 Oct 2023 15:12:14 +0000

My journey with Apache Flink begins with an overview of what Flink actually is.

What better place to start than the Apache Flink website itself:

Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams. Flink has been designed to run in all common cluster environments, perform computations at in-memory speed and at any scale.

Learning Apache Flink S01E01: Where Do I Start?

Fri, 29 Sep 2023 12:57:04 +0000

Like a fortunate child on Christmas Day, I’ve got a brand new toy! A brand new—to me—open-source technology to unwrap, learn, and perhaps even aspire to master elements of within.

I joined Decodable two weeks ago, and since Decodable is built on top of Apache Flink it seems like a great time to learn it. After six years learning Apache Kafka and hearing about this “Flink” thing but—for better or worse—never investigating it, I now have the perfect opportunity to do so.

An Itch That Just Has to Be Scratched… (Or, Why Am I Joining Decodable?)

Thu, 21 Sep 2023 14:25:39 +0000

This week I joined Decodable. I’m grateful to my former colleagues at Treeverse for allowing me to join them on the journey with lakeFS - but something about the streaming world was too strong to resist 😁.

Blog Writing for Developers

Wed, 19 Jul 2023 20:59:09 +0000

Writing is one of the most powerful forms of communication, and it’s useful in a multitude of roles and contexts. As a blog-writing, documentation-authoring, twitter-shitposting DevEx engineer I spend a lot of my time writing. Recently, someone paid me a very nice compliment about a blog I’d written and asked how they could learn to write like me and what resources I’d recommend.

Never one to miss a chance to write and share something, here’s my response to this :)

What Does This DevEx Engineer Do?

Tue, 23 May 2023 23:56:16 +0000

This was originally titled more broadly “What Does A DevEx Engineer Do”, but that made it into a far too tedious and long-winding etymological exploration of the discipline. Instead, I’m going to tell you what this particular instantiation of the entity does 😄

Authoring Wordpress blogs in Markdown (with Google Docs for review)

Wed, 03 May 2023 08:59:17 +0000

Wordpress still, to an extent, rules the blogging world. Its longevity is testament to…something about it ;) However, it’s not my favourite platform in which to write a blog by a long way. It doesn’t support Markdown to the extent that I want. Yes, I’ve tried the plugins; no, they didn’t do what I needed.

I like to write all my content in a structured format - ideally Asciidoc, but I’ll settle for Markdown too. Here’s how I stayed [almost] sane whilst composing a blog in Markdown, reviewing it in Google Docs, and then publishing it in Wordpress in a non-lossy way.

Building Better Docs - Automating Jekyll Builds and Link Checking for PRs

Thu, 20 Apr 2023 08:54:11 +0000

One of the most important ways that a project can help its developers is providing them good documentation. Actually, scratch that. Great documentation.

Using Delta from pySpark - `java.lang.ClassNotFoundException: delta.DefaultSource`

Wed, 05 Apr 2023 15:51:41 +0000

No great insights in this post, just something for folk who Google this error after me and don’t want to waste three hours chasing their tails… 😄

Quickly Convert CSV to Parquet with DuckDB

Tue, 14 Mar 2023 15:12:31 +0000

Here’s a neat little trick you can use with DuckDB to convert a CSV file into a Parquet file:

COPY (SELECT *
	    FROM read_csv('~/data/source.csv',AUTO_DETECT=TRUE))
  TO '~/data/target.parquet' (FORMAT 'PARQUET', CODEC 'ZSTD');

Making the move from Alfred to Raycast

Fri, 03 Mar 2023 23:14:06 +0000

It all started with a tweet.

Aligning mismatched Parquet schemas in DuckDB

Fri, 03 Mar 2023 14:36:08 +0000

What do you do when you want to query over multiple parquet files but the schemas don’t quite line up? Let’s find out 👇🏻

Looking Forwards, and Looking Backwards

Fri, 09 Dec 2022 09:00:00 +0000

As we enter December and 2022 draws to a close, so does a significant chapter in my working career—later this month I’ll be leaving Confluent and onto pastures new.

It’s nearly six years since I wrote a 'moving on' blog entry, and as well as sharing what I’ll be working on next (and why), I also want to reflect on how much I’ve benefited from my time at Confluent and particularly the people with whom I worked.

Data Engineering in 2022: ELT tools

Tue, 08 Nov 2022 19:46:39 +0000

In my quest to bring myself up to date with where the data & analytics engineering world is at nowadays, I’m going to build on my exploration of the storage and access technologies and look at the tools we use for loading and transforming data.

Data Engineering in 2022: Wrangling the feedback data from Current 22 with dbt

Mon, 24 Oct 2022 12:27:14 +0000

I started my dbt journey by poking and pulling at the pre-built jaffle_shop demo running with DuckDB as its data store. Now I want to see if I can put it to use myself to wrangle the session feedback data that came in from Current 2022. I’ve analysed this already, but it struck me that a particular part of it would benefit from some tidying up - and be a good excuse to see what it’s like using dbt to do so.

Data Engineering in 2022: Exploring dbt with DuckDB

Thu, 20 Oct 2022 17:07:04 +0000

I’ve been wanting to try out dbt for some time now, and a recent long-haul flight seemed like the obvious opportunity to do so. Except many of the tutorials with dbt that I found were based on using Cloud, and airplane WiFi is generally sucky or non-existant. Then I found the DuckDB-based demo of dbt, which seemed to fit the bill (🦆 geddit?!) perfectly, since DuckDB runs locally. In addition, DuckDB had appeared on my radar recently and I was keen to check it out.

Current 22 - Session Analysis with DuckDB and Jupyter Notebook

Fri, 14 Oct 2022 16:15:57 +0000

At Current 2022 the audience was given the option to submit ratings. Here’s some analysis I’ve done on the raw data. It’s interesting to poke about it, and it also gave me an excuse to try using DuckDB in a notebook!

Data Engineering in 2022: Architectures & Terminology

Sun, 02 Oct 2022 10:50:56 +0000

This is one of those you had to be there moments. If you come into the world of data and analytics engineering today, ELT is just what it is and is pretty much universally understood. But if you’ve been around for …waves hands… longer than that, you might be confused by what people are calling ELT and ETL. Well, I was ✋.

Current 2022 - 5k Fun Run

Mon, 26 Sep 2022 13:51:42 +0000

At Current 22 a few of us will be going for an early run on Tuesday morning. Everyone is very welcome!

Data Engineering in 2022: Exploring LakeFS with Jupyter and PySpark

Fri, 16 Sep 2022 08:54:45 +0000

With my foray into the current world of data engineering I wanted to get my hands dirty with some of the tools and technologies I’d been reading about. The vehicle for this was trying to understand more about LakeFS, but along the way dabbling with PySpark and S3 (MinIO) too.

I’d forgotten how amazingly useful notebooks are. It’s six years since I wrote about them last (and the last time I tried my hand at PySpark). This blog is basically the notebook, with some more annotations.

Data Engineering: Resources

Wed, 14 Sep 2022 20:57:21 +0000

As I’ve been reading and exploring the current world of data engineering I’ve been adding links to my Raindrop.io collection, so check that out. In addition, below are some specific resources that I’d recommend.

Data Engineering in 2022: Storage and Access

Wed, 14 Sep 2022 17:07:04 +0000

In this article I look at where we store our analytical data, how we organise it, and how we enable access to it. I’m considering here potentially large volumes of data for access throughout an organisation. I’m not looking at data stores that are used for specific purposes (caches, low-latency analytics, graph etc).

The article is part of a series in which I explore the world of data engineering in 2022 and how it has changed from when I started my career in data warehousing 20+ years ago. Read the introduction for more context and background.

Stretching my Legs in the Data Engineering Ecosystem in 2022

Wed, 14 Sep 2022 10:42:30 +0000

For the past 5.5 years I’ve been head-down in the exciting area of stream processing and events, and I realised recently that the world of data and analytics that I worked in up to 2017 which was changing significantly back then (Big Data, y’all!) has evolved and, dare I say it, matured somewhat - and I’ve not necessarily kept up with it. In this series of posts you can follow along as I start to reacquaint myself with where it’s got to these days.

Customising the fields shown in Airtable's Calendar .ics export

Mon, 12 Sep 2022 16:17:08 +0000

Airtable is a rather wonderful tool. It powers the program creation backend process for Kafka Summit and Current. It does, however, have a few frustrating limitations - often where it feels like a feature was built on a Friday afternoon and they didn’t get chance to finish it before knocking off to head to the pub.

Inside the Sausage Factory: How we Built the Program for Current 2022

Wed, 31 Aug 2022 12:14:09 +0000

If you’ve ever been to a conference, particularly as a speaker whose submitted a paper that may or may not have been accepted, you might wonder quite how conferences choose the talks that get accepted.

I had the privilege of chairing the program committee for Current and Kafka Summit this year and curating the final program for both. Here’s a glimpse behind the curtains of how we built the program for Current 2022. It was originally posted as a thread on Twitter.

⚡️ Writing an abstract for a lightning talk ⚡️

Wed, 31 Aug 2022 11:13:38 +0000

(src)

Lightning talks are generally 5-10 minutes. As the name implies - they are quick!

A good lightning talk is not just your breakout talk condensed into a shorter time frame. You can’t simply deliver the same material faster, or the same material at a higher level, or the same material with a few bits left out

How to Write a Good Tech Conference Abstract - Learn from the Mistakes of Others

Wed, 20 Jul 2022 08:58:38 +0000

Building the program for any conference is not an easy task. There will always be a speaker disappointed that their talk didn’t get in—or perhaps an audience who are disappointed that a particular talk did get in. As the chair of the program committee for Current 22 one of the things that I’ve found really useful in building out the program this time round are the comments that the program committee left against submissions as they reviewed them.

There were some common patterns I saw, and I thought it would be useful to share these here. Perhaps you’re an aspiring conference speaker looking to understand what mistakes to avoid. Maybe you’re an existing speaker whose abstracts don’t get accepted as often as you’d like. Or perhaps you’re just curious as to what goes on behind the curtains :)

Remote-First Developer Advocacy

Thu, 07 Apr 2022 21:19:48 +0000

I’m convinced that a developer advocate can be effective remotely. As a profession, we’ve all spent two years figuring out how to do just that. Some of it worked out great. Some of it, less so.

I made the decision during COVID to stop travelling as part of my role as a developer advocate. In this article, I talk about my experience with different areas of advocacy done remotely.

Hanging up my Boarding Passes and Jetlag…for now

Thu, 07 Apr 2022 20:58:33 +0000

I recently started writing an abstract for a conference later this year and realised that I’m not even sure if I want to do it. Not the conference—it’s a great one—but just the whole up on stage doing a talk thing. I can’t work out if this is just nerves from the amount of time off the stage, or something more fundamental to deal with.

Using GitHub Actions to build automagic Hugo previews of draft articles

Wed, 06 Apr 2022 19:30:13 +0000

This blog is written in Asciidoc, built using Hugo, and hosted on GitHub Pages. I recently wanted to share the draft of a post I was writing with someone and ended up exporting a local preview to a PDF - not a great workflow! This blog post shows you how to create an automagic hosted preview of any draft content on Hugo using GitHub Actions.

This is useful for previewing and sharing one’s own content, but also for making good use of GitHub as a collaborative platform - if someone reviews and amends your PR the post gets updated in the preview too.

🏃🚶 The unofficial Kafka Summit London 2022 Run/Walk 🏃🚶

Tue, 05 Apr 2022 09:54:05 +0000

Kafka Summit London IS BACK! After COVID spoiled everyone’s fun and fundamentally screwed everything up for the past two years, I cannot wait to be back at an in-person conference. At the last Kafka Summit in the beforetimes (San Francisco, 2019) some of us got together for a run (or walk) across the GoldenGate bridge. I can’t promise quite the same views, but I thought it would be fun to do something similar when we meet in London later this month.

My Favourite Tools on the Mac (Setting up a new Mac)

Thu, 29 Jul 2021 22:00:08 +0100

This is the software counterpart to my previous article in which I looked at my workstation’s hardware setup. Some of these are unique or best-of-breed, others may have been sherlocked but I stick with them anyway :)

Why I use Alfred App (and maybe you should too)

Thu, 29 Jul 2021 21:24:08 +0100

I’ve used Alfred for years, and it’s one of the first apps I’ll install on a fresh Mac. It’s like the Cmd-Space search integration that MacOS has, but so much more than that. Here’s a few of the really powerful features that makes it the first app I’ll reach for to install on any new Mac - and which it’ll feel like I’m trying to work with one arm tied behind my back if I don’t have :)

A bash script to deploy ksqlDB queries automagically

Thu, 01 Apr 2021 23:06:22 +0100

There’s a bunch of improvements in the works for how ksqlDB handles code deployments and migrations. For now though, for deploying queries there’s the option of using headless mode (which is limited to one query file and disables subsequent interactive work on the server from a CLI), manually running commands (yuck), or using the REST endpoint to deploy queries automagically. Here’s an example of doing that.

Loading CSV data into Confluent Cloud using the FilePulse connector

Fri, 26 Mar 2021 17:25:22 +0000

The FilePulse connector from Florian Hussonnois is a really useful connector for Kafka Connect which enables you to ingest flat files including CSV, JSON, XML, etc into Kafka. You can read more it in its overview here. Other connectors for ingested CSV data include kafka-connect-spooldir (which I wrote about previously), and kafka-connect-fs.

Here I’ll show how to use it to stream CSV data into a topic in Confluent Cloud. You can apply the same config pattern to any other secured Kafka cluster.

Connecting to managed ksqlDB in Confluent Cloud with REST and ksqlDB CLI

Wed, 24 Mar 2021 09:36:43 +0000

Using ksqlDB in Confluent Cloud makes things a whole bunch easier because now you just get to build apps and streaming pipelines, instead of having to run and manage a bunch of infrastructure yourself.

Once you’ve got ksqlDB provisioned on Confluent Cloud you can use the web-based editor to build and run queries. You can also connect to it using the REST API and the ksqlDB CLI tool. Here’s how.

Using ksqlDB to process data ingested from ActiveMQ with Kafka Connect

Fri, 19 Mar 2021 10:30:47 +0000

The ActiveMQ source connector creates a Struct holding the value of the message from ActiveMQ (as well as its key). This is as would be expected. However, you can encounter challenges in working with the data if the ActiveMQ data of interest within the payload is complex. Things like converters and schemas can get really funky, really quick.

Kafka Connect JDBC Sink deep-dive: Working with Primary Keys

Fri, 12 Mar 2021 12:16:16 +0000

The Kafka Connect JDBC Sink can be used to stream data from a Kafka topic to a database such as Oracle, Postgres, MySQL, DB2, etc.

It supports many permutations of configuration around how primary keys are handled. The documentation details these. This article aims to illustrate and expand on this.

Kafka Connect - SQLSyntaxErrorException: BLOB/TEXT column … used in key specification without a key length

Thu, 11 Mar 2021 11:25:57 +0000

I got the error SQLSyntaxErrorException: BLOB/TEXT column 'MESSAGE_KEY' used in key specification without a key length with Kafka Connect JDBC Sink connector (v10.0.2) and MySQL (8.0.23)

Quick profiling of data in Apache Kafka using kafkacat and visidata

Thu, 04 Mar 2021 14:23:15 +0000

ksqlDB is a fantastically powerful tool for processing and analysing streams of data in Apache Kafka. But sometimes, you just want a quick way to profile the data in a topic in Kafka. I wrote about this previously with a convoluted (but effective) set of bash commands pipelined together to perform a GROUP BY on data. Then someone introduced me to visidata, which makes it all a lot quicker!

Using Open Sea Map data in Kibana maps

Thu, 04 Mar 2021 09:23:05 +0000

Kibana’s map functionality is a powerful way to visualise data that has a location element in it. I was recently working with data about ships at sea, and whilst the built in Road map is very good it doesn’t show much maritime detail.

Kibana’s map visualisation has the option to pull in additional visual information from other places (known as tile servers). I found a list of Tile servers, which had details of OpenSeaMap which includes:

Loading delimited data into Kafka - quick & dirty (but effective)

Fri, 26 Feb 2021 22:45:36 +0000

Whilst Apache Kafka is an event streaming platform designed for, well, streams of events, it’s perfectly valid to use it as a store of data which perhaps changes only occasionally (or even never). I’m thinking here of reference data (lookup data) that’s used to enrich regular streams of events.

You might well get your reference data from a database where it resides and do so effectively using CDC - but sometimes it comes down to those pesky CSV files that we all know and love/hate. Simple, awful, but effective. I wrote previously about loading CSV data into Kafka from files that are updated frequently, but here I want to look at CSV files that are not changing. Kafka Connect simplifies getting data in to (and out of) Kafka but even Kafka Connect becomes a bit of an overhead when you just have a single file that you want to load into a topic and then never deal with again. I spent this afternoon wrangling with a couple of CSV-ish files, and building on my previous article about neat tricks you can do in bash with data, I have some more to share with you here :)

📼 ksqlDB HOWTO - A mini video series 📼

Wed, 17 Feb 2021 23:12:33 +0000

Some people learn through doing - and for that there’s a bunch of good ksqlDB tutorials here and here. Others may prefer to watch and listen first, before getting hands on. And for that, I humbly offer you this little series of videos all about ksqlDB. They’re all based on a set of demo scripts that you can run for yourself and try out.

🚨 Make sure you subscribe to my YouTube channel so that you don’t miss more videos like these!

Performing a GROUP BY on data in bash

Tue, 02 Feb 2021 17:23:21 +0000

One of the fun things about working with data over the years is learning how to use the tools of the day—but also learning to fall back on the tools that are always there for you - and one of those is bash and its wonderful library of shell tools.

There’s an even better way than I’ve described here, and it’s called visidata. I’ve written about it more over here.

I’ve been playing around with a new data source recently, and needed to understand more about its structure. Within a single stream there were multiple message types.

Running as root on Docker images that don't use root

Wed, 13 Jan 2021 12:13:41 +0000

tl;dr: specify the --user root argument:

docker exec --interactive \
            --tty \
            --user root \
            --workdir / \
            container-name bash

Running a self-managed Kafka Connect worker for Confluent Cloud

Mon, 11 Jan 2021 17:02:03 +0000

Confluent Cloud is not only a fully-managed Apache Kafka service, but also provides important additional pieces for building applications and pipelines including managed connectors, Schema Registry, and ksqlDB. Managed Connectors are run for you (hence, managed!) within Confluent Cloud - you just specify the technology to which you want to integrate in or out of Kafka and Confluent Cloud does the rest.

Creating topics with Kafka Connect

Wed, 06 Jan 2021 12:18:51 +0000

When Kafka Connect ingests data from a source system into Kafka it writes it to a topic. If you have set auto.create.topics.enable = true on your broker then the topic will be created when written to. If auto.create.topics.enable = false (as it is on Confluent Cloud and many self-managed environments, for good reasons) then you can tell Kafka Connect to create those topics first. This was added in Apache Kafka 2.6 (Confluent Platform 6.0) - prior to that you had to manually create the topics yourself otherwise the connector would fail.

Kafka Connect - Deep Dive into Single Message Transforms

Mon, 04 Jan 2021 14:26:40 +0000

KIP-66 was added in Apache Kafka 0.10.2 and brought new functionality called Single Message Transforms (SMT). Using SMT you can modify the data and its characteristics as it passes through Kafka Connect pipeline, without needing additional stream processors. For things like manipulating fields, changing topic names, conditionally dropping messages, and more, SMT are a perfect solution. If you get to things like aggregation, joining streams, and lookups then SMT may not be the best for you and you should head over to Kafka Streams or ksqlDB instead.

🎄 Twelve Days of SMT 🎄 - Day 12: Community Transformations

Wed, 23 Dec 2020 16:00:00 +0000

Apache Kafka ships with many Single Message Transformations (SMT) included - but the great thing about it being an open API is that people can, and do, write their own transformations. Many of these are shared with the wider community, and in this final installment of the series I’m going to look at some of the transformations written by Jeremy Custenborder and available in kafka-connect-transform-common which can be downloaded and installed from Confluent Hub (or built from source, if you like that kind of thing). Also check out the XML transformation by the same author, which I’ve written about previously.

🎄 Twelve Days of SMT 🎄 - Day 11: Predicate and Filter

Tue, 22 Dec 2020 16:00:00 +0000

Apache Kafka 2.6 included KIP-585 which adds support for defining predicates against which transforms are conditionally executed, as well as a Filter Single Message Transform to drop messages - which in combination means that you can conditionally drop messages.

As part of Apache Kafka, Kafka Connect ships with pre-built Single Message Transforms and Predicates, but you can also write you own. The API for each is documented: Transformation / Predicate. The predicates that ship with Apache Kafka are:

RecordIsTombstone - The value part of the message is null (denoting a tombstone message)
HasHeaderKey- Matches if a header exists with the name given
TopicNameMatches - Matches based on topic

🎄 Twelve Days of SMT 🎄 - Day 10: ReplaceField

Mon, 21 Dec 2020 16:00:00 +0000

The ReplaceField Single Message Transform has three modes of operation on fields of data passing through Kafka Connect:

Include only the fields specified in the list (include)
Include all fields except the ones specified (exclude)
Rename field(s) (renames)

Scheduling Hugo Builds on GitHub pages with GitHub Actions

Sun, 20 Dec 2020 23:45:03 +0000

Over the years I’ve used various blogging platforms; after a brief dalliance with Blogger I started for real with the near-inevitable Wordpress.com. From there I decided it would be fun to self-host using Ghost, and then almost exactly two years ago to the day decided it definitely was not fun to spend time patching and upgrading my blog platform instead of writing blog articles, so headed over to my current platform of choice: Hugo hosted on GitHub pages. This has worked extremely well for me during that time, doing everything I want from it until recently.

🎄 Twelve Days of SMT 🎄 - Day 9: Cast

Fri, 18 Dec 2020 16:00:00 +0000

The Cast Single Message Transform lets you change the data type of fields in a Kafka message, supporting numerics, string, and boolean.

🎄 Twelve Days of SMT 🎄 - Day 8: TimestampConverter

Thu, 17 Dec 2020 16:00:00 +0000

The TimestampConverter Single Message Transform lets you work with timestamp fields in Kafka messages. You can convert a string into a native Timestamp type (or Date or Time), as well as Unix epoch - and the same in reverse too.

This is really useful to make sure that data ingested into Kafka is correctly stored as a Timestamp (if it is one), and also enables you to write a Timestamp out to a sink connector in a string format that you choose.

🎄 Twelve Days of SMT 🎄 - Day 7: TimestampRouter

Wed, 16 Dec 2020 16:00:00 +0000

Just like the RegExRouter, the TimeStampRouter can be used to modify the topic name of messages as they pass through Kafka Connect. Since the topic name is usually the basis for the naming of the object to which messages are written in a sink connector, this is a great way to achieve time-based partitioning of those objects if required. For example, instead of streaming messages from Kafka to an Elasticsearch index called cars, they can be routed to monthly indices e.g. cars_2020-10, cars_2020-11, cars_2020-12, etc.

The TimeStampRouter takes two arguments; the format of the final topic name to generate, and the format of the timestamp to put in the topic name (based on SimpleDateFormat).

"transforms"                                     : "addTimestampToTopic",
"transforms.addTimestampToTopic.type"            : "org.apache.kafka.connect.transforms.TimestampRouter",
"transforms.addTimestampToTopic.topic.format"    : "${topic}_${timestamp}",
"transforms.addTimestampToTopic.timestamp.format": "YYYY-MM-dd"

🎄 Twelve Days of SMT 🎄 - Day 6: InsertField II

Tue, 15 Dec 2020 16:00:00 +0000

We kicked off this series by seeing on day 1 how to use InsertField to add in the timestamp to a message passing through the Kafka Connect sink connector. Today we’ll see how to use the same Single Message Transform to add in a static field value, as well as the name of the Kafka topic, partition, and offset from which the message has been read.

"transforms"                                : "insertStaticField1",
"transforms.insertStaticField1.type"        : "org.apache.kafka.connect.transforms.InsertField$Value",
"transforms.insertStaticField1.static.field": "sourceSystem",
"transforms.insertStaticField1.static.value": "NeverGonna"

🎄 Twelve Days of SMT 🎄 - Day 5: MaskField

Mon, 14 Dec 2020 16:00:00 +0000

If you want to mask fields of data as you ingest from a source into Kafka, or write to a sink from Kafka with Kafka Connect, the MaskField Single Message Transform is perfect for you. It retains the fields whilst replacing its value.

To use the Single Message Transform you specify the field to mask, and its replacement value. To mask the contents of a field called cc_num you would use:

"transforms"                               : "maskCC",
"transforms.maskCC.type"                   : "org.apache.kafka.connect.transforms.MaskField$Value",
"transforms.maskCC.fields"                 : "cc_num",
"transforms.maskCC.replacement"            : "****-****-****-****"

🎄 Twelve Days of SMT 🎄 - Day 4: RegExRouter

Fri, 11 Dec 2020 16:40:18 +0000

If you want to change the topic name to which a source connector writes, or object name that’s created on a target by a sink connector, the RegExRouter is exactly what you need.

To use the Single Message Transform you specify the pattern in the topic name to match, and its replacement. To drop a prefix of test- from a topic you would use:

"transforms"                             : "dropTopicPrefix",
"transforms.dropTopicPrefix.type"        : "org.apache.kafka.connect.transforms.RegexRouter",
"transforms.dropTopicPrefix.regex"       : "test-(.*)",
"transforms.dropTopicPrefix.replacement" : "$1"

🎄 Twelve Days of SMT 🎄 - Day 3: Flatten

Thu, 10 Dec 2020 16:25:00 +0000

The Flatten Single Message Transform (SMT) is useful when you need to collapse a nested message down to a flat structure.

To use the Single Message Transform you only need to reference it; there’s no additional configuration required:

"transforms"                    : "flatten",
"transforms.flatten.type"       : "org.apache.kafka.connect.transforms.Flatten$Value"

🎄 Twelve Days of SMT 🎄 - Day 2: ValueToKey and ExtractField

Wed, 09 Dec 2020 20:00:18 +0000

Setting the key of a Kafka message is important as it ensures correct logical processing when consumed across multiple partitions, as well as being a requirement when joining to messages in other topics. When using Kafka Connect the connector may already set the key, which is great. If not, you can use these two Single Message Transforms (SMT) to set it as part of the pipeline based on a field in the value part of the message.

To use the ValueToKey Single Message Transform specify the name of the field (id) that you want to copy from the value to the key:

"transforms"                    : "copyIdToKey",
"transforms.copyIdToKey.type"   : "org.apache.kafka.connect.transforms.ValueToKey",
"transforms.copyIdToKey.fields" : "id",

🎄 Twelve Days of SMT 🎄 - Day 1: InsertField (timestamp)

Tue, 08 Dec 2020 22:23:18 +0000

You can use the InsertField Single Message Transform (SMT) to add the message timestamp into each message that Kafka Connect sends to a sink.

To use the Single Message Transform specify the name of the field (timestamp.field) that you want to add to hold the message timestamp:

"transforms"                         : "insertTS",
"transforms.insertTS.type"           : "org.apache.kafka.connect.transforms.InsertField$Value",
"transforms.insertTS.timestamp.field": "messageTS"

Life as a Developer Advocate, nine months into a pandemic

Thu, 03 Dec 2020 22:15:59 +0100

Back in March 2020 the western world came to somewhat of a juddering halt, thanks to COVID-19. No-one knew then what would happen, but there was the impression that whilst the next few months were a write-off for sure, maybe things would pick up again later in the year.

It’s now early December 2020, and nothing is picking up any time soon. Summer provided a respite from the high levels of infection and mortality (in the UK at least), but then numbers spiked again in many places around the world and what was punted down the river back in March is being firmly punted yet again now.

My Workstation - 2020

Wed, 02 Dec 2020 17:19:21 +0100

Is a blog even a blog nowadays if it doesn’t include a "Here is my home office setup"?

Thanks to conferences all being online, and thus my talks being delivered from my study—and my habit of posting a #SpeakerSelfie each time I do a conference talk—I often get questions about my setup. Plus, I’m kinda pleased with it so I want to show it off too ;-)

Keynote - Why is Replace Fonts greyed out?

Fri, 13 Nov 2020 15:49:37 +0000

Very short & sweet this post, but Google turned up nothing when I was stuck so hopefully I’ll save someone else some head scratching by sharing this.

Kafka Connect, ksqlDB, and Kafka Tombstone messages

Tue, 03 Nov 2020 17:14:33 +0000

As you may already realise, Kafka is not just a fancy message bus, or a pipe for big data. It’s an event streaming platform! If this is news to you, I’ll wait here whilst you read this or watch this…

Streaming Geopoint data from Kafka to Elasticsearch

Tue, 03 Nov 2020 10:36:18 +0000

Streaming data from Kafka to Elasticsearch is easy with Kafka Connect - you can see how in this tutorial and video.

One of the things that sometimes causes issues though is how to get location data correctly indexed into Elasticsearch as geo_point fields to enable all that lovely location analysis. Unlike data types like dates and numerics, Elasticsearch’s Dynamic Field Mapping won’t automagically pick up geo_point data, and so you have to do two things:

ksqlDB - How to model a variable number of fields in a nested value (`STRUCT`)

Wed, 07 Oct 2020 11:44:51 +0100

There was a good question on StackOverflow recently in which someone was struggling to find the appropriate ksqlDB DDL to model a source topic in which there was a variable number of fields in a STRUCT.

Streaming XML messages from IBM MQ into Kafka into MongoDB

Mon, 05 Oct 2020 10:09:41 +0100

Let’s imagine we have XML data on a queue in IBM MQ, and we want to ingest it into Kafka to then use downstream, perhaps in an application or maybe to stream to a NoSQL store like MongoDB.

This same pattern for ingesting XML will work with other connectors such as JMS and ActiveMQ.

Ingesting XML data into Kafka - Option 3: Kafka Connect FilePulse connector

Thu, 01 Oct 2020 15:09:41 +0100

👉 Ingesting XML data into Kafka - Introduction

We saw in the first post how to hack together an ingestion pipeline for XML into Kafka using a source such as curl piped through xq to wrangle the XML and stream it into Kafka using kafkacat, optionally using ksqlDB to apply and register a schema for it.

The second one showed the use of any Kafka Connect source connector plus the kafka-connect-transform-xml Single Message Transformation. Now we’re going to take a look at a source connector from the community that can also be used to ingest XML data into Kafka.

Ingesting XML data into Kafka - Option 2: Kafka Connect plus Single Message Transform

Thu, 01 Oct 2020 14:09:41 +0100

We previously looked at the background to getting XML into Kafka, and potentially how [not] to do it. Now let’s look at the proper way to build a streaming ingestion pipeline for XML into Kafka, using Kafka Connect.

If you’re unfamiliar with Kafka Connect, check out this quick intro to Kafka Connect here. Kafka Connect’s excellent plugable architecture means that we can pair any source connector to read XML from wherever we have it (for example, a flat file, or a MQ, or anywhere else), with a Single Message Transform to transform the XML into a payload with a schema, and finally a converter to serialise the data in a form that we would like to use such as Avro or Protobuf.

Ingesting XML data into Kafka - Option 1: The Dirty Hack

Thu, 01 Oct 2020 13:09:41 +0100

👉 Ingesting XML data into Kafka - Introduction

What would a blog post on rmoff.net be if it didn’t include the dirty hack option? 😁

The secret to dirty hacks is that they are often rather effective and when needs must, they can suffice. If you’re prototyping and need to JFDI, a dirty hack is just fine. If you’re looking for code to run in Production, then a dirty hack probably is not fine.

Ingesting XML data into Kafka - Introduction

Thu, 01 Oct 2020 12:09:41 +0100

XML has been around for 20+ years, and whilst other ways of serialising our data have gained popularity in more recent times (such as JSON, Avro, and Protobuf), XML is not going away soon. Part of that is down to technical reasons (clearly defined and documented schemas), and part of it is simply down to enterprise inertia - having adopted XML for systems in the last couple of decades, they’re not going to be changing now just for some short-term fad. See also COBOL.

`abcde` - Error trying to calculate disc ids without lead-out information

Thu, 01 Oct 2020 09:16:11 +0100

Short & sweet to help out future Googlers. Trying to use abcde I got the error:

[WARNING] something went wrong while querying the CD... Maybe a DATA CD or the CD is not loaded?
[WARNING] Error trying to calculate disc ids without lead-out information.

IBM MQ on Docker - Channel was blocked

Thu, 01 Oct 2020 01:09:41 +0100

Running IBM MQ in a Docker container and the client connecting to it was throwing repeated Channel was blocked errors.

Setting key value when piping from jq to kafkacat

Wed, 30 Sep 2020 20:54:09 +0100

One of my favourite hacks for getting data into Kafka is using kafkacat and stdin, often from jq. You can see this in action with Wi-Fi data, IoT data, and data from a REST endpoint. This is fine for getting values into a Kafka message - but Kafka messages are key/value, and being able to specify a key is can often be important.

Here’s a way to do that, using a separator and some jq magic. Note that at the moment kafkacat only supports single byte separator characters, so you need to choose carefully. If you pick a separator that also appears in your data, it’s possibly going to have unintended consequences.

Some of my favourite public data sets

Fri, 25 Sep 2020 12:09:41 +0100

Readers of a certain age and RDBMS background will probably remember northwind, or HR, or OE databases - or quite possibly not just remember them but still be using them. Hardcoded sample data is fine, and it’s great for repeatable tutorials and examples - but it’s boring as heck if you want to build an example with something that isn’t using the same data set for the 100th time.

📌 🎁 A collection of Kafka-related talks 💝

Wed, 23 Sep 2020 15:00:05 +0100

Here’s a collection of Kafka-related talks, just for you.

Each one has 🍿🎥 a recording, 📔 slides, and 👾 code to go and try out.

Using the Debezium MS SQL connector with ksqlDB embedded Kafka Connect

Fri, 18 Sep 2020 10:00:05 +0100

Prompted by a question on StackOverflow I thought I’d take a quick look at setting up ksqlDB to ingest CDC events from Microsoft SQL Server using Debezium. Some of this is based on my previous article, Streaming data from SQL Server to Kafka to Snowflake ❄️ with Kafka Connect.

Setting up the Docker Compose

I like standalone, repeatable, demo code. For that reason I love using Docker Compose and I embed everything in there - connector installation, the kitchen sink - the works.

Including content from external links with Asciidoc in Hugo

Fri, 18 Sep 2020 09:00:05 +0100

I use Hugo for my blog, hosted on GitHub pages. One of the reasons I’m really happy with it is that I can use Asciidoc to author my posts. I was writing a blog recently in which I wanted to include some code that’s hosted on GitHub. I could have copied & pasted it into the blog but that would be lame!

With Asciidoc you can use the include:: directive to include both local files:

What is Kafka Connect?

Fri, 11 Sep 2020 16:00:05 +0100

Kafka Connect is the integration API for Apache Kafka. Check out this video for an overview of what Kafka Connect enables you to do, and how to do it.

Counting the number of messages in a Kafka topic

Tue, 08 Sep 2020 10:00:05 +0100

There’s ways, and then there’s ways, to count the number of records/events/messages in a Kafka topic. Most of them are potentially inaccurate, or inefficient, or both. Here’s one that falls into the potentially inefficient category, using kafkacat to read all the messages and pipe to wc which with the -l will tell you how many lines there are, and since each message is a line, how many messages you have in the Kafka topic:

$ kafkacat -b broker:29092 -t mytestopic -C -e -q| wc -l
       3

Poking around the search engines in Google Chrome

Mon, 07 Sep 2020 23:00:05 +0100

Google Chrome automagically adds sites that you visit which support searching to a list of custom search engines. For each one you can set a keyword which activates it, so based on the above list if I want to search Amazon I can just type a <tab> and then my search term

🤖Building a Telegram bot with Apache Kafka, Go, and ksqlDB

Thu, 20 Aug 2020 10:00:05 +0100

I had the pleasure of presenting at DataEngBytes recently, and am delighted to share with you the 🗒️ slides, 👾 code, and 🎥 recording of my ✨brand new talk✨:

🤖Building a Telegram bot with Apache Kafka, Go, and ksqlDB

Telegram bot - BOT_COMMAND_INVALID

Thu, 23 Jul 2020 15:00:05 +0100

A tiny snippet since I wasted 10 minutes going around the houses on this one…

tl;dr: If you try to create a command that is not in lower case (e.g. Alert not alert) then the setMyCommands API will return BOT_COMMAND_INVALID

Learning Golang (some rough notes) - S02E09 - Processing chunked responses before EOF is reached

Thu, 23 Jul 2020 10:00:05 +0100

The server sends Transfer-Encoding: chunked data, and you want to work with the data as you get it, instead of waiting for the server to finish, the EOF to fire, and then process the data?

Learning Golang (some rough notes) - S02E08 - Checking Kafka advertised.listeners with Go

Fri, 17 Jul 2020 17:00:05 +0100

At the beginning of all this my aim was to learn something new (Go), and use it to write a version of a utility that I’d previously hacked together in Python that checks your Apache Kafka broker configuration for possible problems with the infamous advertised.listeners setting. Check out a blog that I wrote which explains all about Apache Kafka and listener configuration.

You can find the code at https://github.com/rmoff/kafka-listeners

Learning Golang (some rough notes) - S02E07 - Splitting Go code into separate source files and building a binary executable

Thu, 16 Jul 2020 11:00:05 +0100

So far I’ve been running all my code either in the Go Tour sandbox, using Go Playground, or from a single file in VS Code. My explorations in the previous article ended up with a a source file that was starting to get a little bit unwieldily, so let’s take a look at how that can be improved.

Within my most recent code, I have the main function and the doProduce function, which is fine when collapsed down:

Learning Golang (some rough notes) - S02E06 - Putting the Producer in a function and handling errors in a Go routine

Wed, 15 Jul 2020 14:00:05 +0100

When I set out to learn Go one of the aims I had in mind was to write a version of this little Python utility which accompanies a blog I wrote recently about understanding and diagnosing problems with Kafka advertised listeners. Having successfully got Producer, Consumer, and AdminClient API examples working, it is now time to turn to that task.

Learning Golang (some rough notes) - S02E05 - Kafka Go AdminClient

Wed, 15 Jul 2020 11:00:05 +0100

Having ticked off the basics with an Apache Kafka producer and consumer in Go, let’s now check out the AdminClient. This is useful for checking out metadata about the cluster, creating topics, and stuff like that.

Learning Golang (some rough notes) - S02E04 - Kafka Go Consumer (Function-based)

Tue, 14 Jul 2020 13:59:05 +0100

Last time I looked at creating my first Apache Kafka consumer in Go, which used the now-deprecated channel-based consumer. Whilst idiomatic for Go, it has some issues which mean that the function-based consumer is recommended for use instead. So let’s go and use it!

Instead of reading from the Events() channel of the consumer, we read events using the Poll() function with a timeout. The way we handle events (a switch based on their type) is the same:

Learning Golang (some rough notes) - S02E03 - Kafka Go Consumer (Channel-based)

Tue, 14 Jul 2020 11:59:05 +0100

Having written my first Kafka producer in Go, and even added error handling to it, the next step was to write a consumer. It follows closely the pattern of Producer code I finished up with previously, using the channel-based approach for the Consumer:

Learning Golang (some rough notes) - S02E02 - Adding error handling to the Producer

Fri, 10 Jul 2020 10:59:05 +0100

I looked last time at the very bare basics of writing a Kafka producer using Go. It worked, but only with everything lined up and pointing the right way. There was no error handling of any sorts. Let’s see about fixing this now.

Learning Golang (some rough notes) - S02E01 - My First Kafka Go Producer

Wed, 08 Jul 2020 17:59:05 +0100

Learning Golang (some rough notes) - S02E00 - Kafka and Go

Wed, 08 Jul 2020 10:59:05 +0100

With the first leg of my journey with Go done (starting from a very rudimentary base), the next step for me was to bring it into my current area of interest and work - Apache Kafka.

Learning Golang (some rough notes) - S01E10 - Concurrency (Web Crawler)

Fri, 03 Jul 2020 16:59:05 +0100

👉 A Tour of Go : sync.Mutex

In the previous exercise I felt my absence of a formal CompSci background with the introduction of Binary Sorted Trees, and now I am concious of it again with learning about mutex. I’d heard of them before, mostly when Oracle performance folk were talking about wait types - TIL it stands for mutual exclusion!

Why JSON isn't the same as JSON Schema in Kafka Connect converters and ksqlDB (Viewing Kafka messages bytes as hex)

Fri, 03 Jul 2020 08:16:36 +0100

I’ve been playing around with the new SerDes (serialisers/deserialisers) that shipped with Confluent Platform 5.5 - Protobuf, and JSON Schema (these were added to the existing support for Avro). The serialisers (and associated Kafka Connect converters) take a payload and serialise it into bytes for sending to Kafka, and I was interested in what those bytes look like. For that I used my favourite Kafka swiss-army knife: kafkacat.

Learning Golang (some rough notes) - S01E09 - Concurrency (Channels, Goroutines)

Thu, 02 Jul 2020 16:59:05 +0100

A Tour of Go : Goroutines was OK but as with some previous material I headed over to Go by example for clearer explanations.

Learning Golang (some rough notes) - S01E08 - Images

Thu, 02 Jul 2020 14:59:05 +0100

👉 A Tour of Go : Exercise: Images

This is based on the Picture generator from the Slices exercise.

Learning Golang (some rough notes) - S01E07 - Readers

Wed, 01 Jul 2020 15:59:05 +0100

👉 A Tour of Go : Readers

I’m not intending to pick holes in the Tour…but it’s not helping itself ;-)

For an introductory text, it makes a ton of assumptions about the user. Here it introduces Readers, and the explanation is good—but the example code looks like this:

Learning Golang (some rough notes) - S01E06 - Errors

Wed, 01 Jul 2020 10:59:05 +0100

👉 A Tour of Go : Exercise: Errors

Like Interfaces, the Tour didn’t really do it for me on Errors either. Too absract, and not enough explanation of the code examples for my liking. It also doesn’t cover the errors package which other tutorial do. I’m not clear if that’s because the errors package isn’t used much, or the Tour focusses only on teaching the raw basics.

Learning Golang (some rough notes) - S01E05 - Interfaces

Tue, 30 Jun 2020 16:59:05 +0100

👉 A Tour of Go : Interfaces

This page really threw me, for several reasons:

The text notes that there’s an error (so why don’t they fix it?)
The provided code doesn’t run (presumably because of the above error)
It’s not clear if this is a deliberate error to illustrate a point, or just a snafu

Learning Golang (some rough notes) - S01E04 - Function Closures

Mon, 29 Jun 2020 14:59:05 +0100

👉 A Tour of Go : Function Closures

So far the Tour has been 🤔 and 🧐 and even 🤨 but function closures had me 🤯 …

Each of the words on the page made sense but strung together in a sentence didn’t really make any sense to me.

Learning Golang (some rough notes) - S01E03 - Maps

Mon, 29 Jun 2020 13:59:05 +0100

👉 A Tour of Go : Exercise - Maps

Implement WordCount

This is probably bread-and-butter for any seasoned programmer, but I enjoyed the simple process and satisfaction of breaking the problem down into steps to solve using what the tutorial had just covered. Sketching out the logic in pseudo-code first, I figured that I wanted to do this:

Learning Golang (some rough notes) - S01E02 - Slices

Thu, 25 Jun 2020 11:20:23 +0100

Learning Go : Background

👉 A Tour of Go : Slices

Slices made sense, until I got to Slice length and capacity. Two bits puzzled me in this code:

Learning Golang (some rough notes) - S01E01 - Pointers

Thu, 25 Jun 2020 11:15:23 +0100

Learning Go : Background

👉 A Tour of Go : Pointers

I’ve never used pointers before. Found plenty of good resources about what they are, e.g.

But why? It’s like explaining patiently to someone that 2+2 = 4, without really explaining why would we want to add two numbers together in the first place.

Learning Golang (some rough notes) - S01E00

Thu, 25 Jun 2020 11:13:23 +0100

My background is not a traditional CompSci one. I studied Music at university, and managed to wangle my way into IT through various means, ending up doing what I do now with no formal training in coding, and a grab-bag of hacky programming attempts on my CV. My weapons of choice have been BBC Basic, VBA, ASP, and more recently some very unpythonic-Python. It’s got me by, but I figured recently I’d like to learn something new, and several people pointed to Go as a good option.

How to install connector plugins in Kafka Connect

Fri, 19 Jun 2020 17:28:09 +0100

Kafka Connect (which is part of Apache Kafka) supports pluggable connectors, enabling you to stream data between Kafka and numerous types of system, including to mention just a few:

Databases
Message Queues
Flat files
Object stores

The appropriate plugin for the technology which you want to integrate can be found on Confluent Hub.

Loading CSV data into Kafka

Wed, 17 Jun 2020 17:57:18 +0100

For whatever reason, CSV still exists as a ubiquitous data interchange format. It doesn’t get much simpler: chuck some plaintext with fields separated by commas into a file and stick .csv on the end. If you’re feeling helpful you can include a header row with field names in.

order_id,customer_id,order_total_usd,make,model,delivery_city,delivery_company,delivery_address
1,535,190899.73,Dodge,Ram Wagon B350,Sheffield,DuBuque LLC,2810 Northland Avenue
2,671,33245.53,Volkswagen,Cabriolet,Edinburgh,Bechtelar-VonRueden,1 Macpherson Crossing

In this article we’ll see how to load this CSV data into Kafka, without even needing to write any code

How to list and create Kafka topics using the REST Proxy API

Fri, 05 Jun 2020 09:46:06 +0100

In v5.5 of Confluent Platform the REST Proxy added new Admin API capabilities, including functionality to list, and create, topics on your cluster.

Check out the docs here and download Confluent Platform here. The REST proxy is Confluent Community Licenced.

Working with JSON nested arrays in ksqlDB - example

Tue, 26 May 2020 10:02:48 +0100

Question from the Confluent Community Slack group:

How can I access the data in object in an array like below using ksqlDB stream


"Total": [
        {
          "TotalType": "Standard",
          "TotalAmount": 15.99
        },
{
          "TotalType": "Old Standard",
          "TotalAmount": 16,
" STID":56
        }
]

Searching Alfred's Clipboard history programatically

Mon, 18 May 2020 12:46:02 +0100

Alfred is one of my favourite productivity apps for the Mac. It’s a file indexer, a clipboard manager, a snippet expander - and that’s just scratching the surface really. I recently got a new machine without it installed and realised just how much I rely on Alfred, particularly its clipboard manager.

Building a Telegram bot with Apache Kafka and ksqlDB

Mon, 18 May 2020 11:28:15 +0100

Imagine you’ve got a stream of data; it’s not “big data,” but it’s certainly a lot. Within the data, you’ve got some bits you’re interested in, and of those bits, you’d like to be able to query information about them at any point. Sounds fun, right?

What if you didn’t need any datastore other than Apache Kafka itself to be able to do this? What if you could ingest, filter, enrich, aggregate, and query data with just Kafka? With ksqlDB we can do just this, and I want to show you exactly how, using a Telegram bot as the application looking up state from the inbound stream of events:

Add Markers list from Screenflow to Youtube Table of Contents

Mon, 04 May 2020 10:20:10 +0100

Screenflow has a useful Markers feature for adding notes to the timeline.

You can use these to helpfully add a table of contents to your Youtube video, but unfortunately Screenflow doesn’t have the option to export them directly. Instead, use the free Subler program as an intermediary (download it from here).

Export from Screenflow with a chapters track
Open the file in Subler and export to text file

From there, tidy up the text file from the source

Using Confluent Cloud when there is no Cloud (or internet)

Mon, 20 Apr 2020 13:55:46 +0100

☁️Confluent Cloud is a great solution for a hosted and managed Apache Kafka service, with the additional benefits of Confluent Platform such as ksqlDB and managed Kafka Connect connectors. But as a developer, you won’t always have a reliable internet connection. Train, planes, and automobiles—not to mention crappy hotel or conference Wi-Fi. Wouldn’t it be useful if you could have a replica of your Cloud data on your local machine? That just pulled down new data automagically, without needing to be restarted each time you got back on the network?

How to install kafkacat on Fedora

Mon, 20 Apr 2020 10:25:32 +0100

kafkacat is one of my go-to tools when working with Kafka. It’s a producer and consumer, but also a swiss-army knife of debugging and troubleshooting capabilities. So when I built a new Fedora server recently, I needed to get it installed. Unfortunately there’s no pre-packed install available on yum, so here’s how to do it manually.

kafkacat is now known as kcat (ref). When invoking the command you will need to use kcat in place of kafkacat.

Converting from AsciiDoc to Google Docs and MS Word

Thu, 16 Apr 2020 14:27:50 +0100

Updated 16 April 2020 to cover formatting tricks & add import to Google Docs info

Short and sweet this one. I’ve written in the past how I love Markdown but I’ve actually moved on from that and now firmly throw my hat in the AsciiDoc ring. I’ll write another post another time explaining why in more detail, but in short it’s just more powerful whilst still simple and readable without compilation.

So anyway, I use AsciiDoc (adoc) for all my technical (and often non-technical) writing now, and from there usually dump it out to HTML which I can share with people as needed:

A quick and dirty way to monitor data arriving on Kafka

Thu, 16 Apr 2020 00:51:18 +0100

I’ve been poking around recently with capturing Wi-Fi packet data and streaming it into Apache Kafka, from where I’m processing and analysing it. Kafka itself is rock-solid - because I’m using ☁️Confluent Cloud and someone else worries about provisioning it, scaling it, and keeping it running for me. But whilst Kafka works just great, my side of the setup—tshark running on a Raspberry Pi—is less than stable. For whatever reason it sometimes stalls and I have to restart the Raspberry Pi and restart the capture process.

Are Tech Conferences Dead?

Fri, 13 Mar 2020 22:19:16 +0000

🦠COVID-19 has well and truly hit the tech scene this week. As well as being full of "WFH tips" for all the tech workers suddenly banished from their offices, my particular Twitter bubble is full of DevRel folk musing and debating about what this interruption means to our profession. For sure, in the short term, the Spring conference season is screwed— all the conferences are cancelled (or postponed).

But what about the future? No-one would ever want to take such a forced hiatus but what an excellent opportunity it is to take a step back and consider why we’re doing what we’re doing - and if we should go back to business as usual once things calm down.

Streaming Wi-Fi trace data from Raspberry Pi to Apache Kafka with Confluent Cloud

Wed, 11 Mar 2020 11:58:13 +0000

Wi-fi is now ubiquitous in most populated areas, and the way the devices communicate leaves a lot of 'digital exhaust'. Usually a computer will have a Wi-Fi device that’s configured to connect to a given network, but often these devices can be configured instead to pick up the background Wi-Fi chatter of surrounding devices.

There are good reasons—and bad—for doing this. Just like taking apart equipment to understand how it works teaches us things, so being able to dissect and examine protocol traffic lets us learn about this. However, by collecting this type of traffic it can be possible to track and analyse behaviour in ways that we may or may not feel comfortable with. Improving public transport? Sure. Tracking shopper behaviour? Meh, less sure.

Kafka Connect JDBC Sink - setting the key field name

Tue, 25 Feb 2020 14:37:12 +0100

I wanted to get some data from a Kafka topic:

ksql> PRINT PERSON_STATS FROM BEGINNING;
Key format: KAFKA (STRING)
Value format: AVRO
rowtime: 2/25/20 1:12:51 PM UTC, key: robin, value: {"PERSON": "robin",
 "LOCATION_CHANGES":1, "UNIQUE_LOCATIONS": 1}

into Postgres, so did the easy thing and used Kafka Connect with the JDBC Sink connector.

Adventures in the Cloud, Part 94: ECS

Thu, 13 Feb 2020 00:12:23 +0000

My name’s Robin, and I’m a Developer Advocate. What that means in part is that I build a ton of demos, and Docker Compose is my jam. I love using Docker Compose for the same reasons that many people do:

Spin up and tear down fully-functioning multi-component environments with ease. No bespoke builds, no cloning of VMs to preserve "that magic state where everything works"
Repeatability. It’s the same each time.
Portability. I can point someone at a docker-compose.yml that I’ve written and they can run the same on their machine with the same results almost guaranteed.

Primitive Keys in ksqlDB

Fri, 07 Feb 2020 10:58:06 +0000

ksqlDB 0.7 will add support for message keys as primitive data types beyond just STRING (which is all we’ve had to date). That means that Kafka messages are going to be much easier to work with, and require less wrangling to get into the form in which you need them. Take an example of a database table that you’ve ingested into a Kafka topic, and want to join to a stream of events. Previously you’d have had to take the Kafka topic into which the table had been ingested and run a ksqlDB processor to re-key the messages such that ksqlDB could join on them. Friends, I am here to tell you that this is no longer needed!

Fantastical / Mac Calendar not showing Google Shared Calendar

Fri, 24 Jan 2020 11:50:01 +0000

Very simple to fix: go to https://calendar.google.com/calendar/syncselect and select the calendars that you want. Click save.

Notes on getting data into InfluxDB from Kafka with Kafka Connect

Thu, 23 Jan 2020 12:01:35 +0000

You can download the InfluxDB connector for Kafka Connect here. Documentation for it is here.

When a message from your source Kafka topic is written to InfluxDB the InfluxDB values are set thus:

Timestamp is taken from the Kafka message timestamp (which is either set by your producer, or the time at which it was received by the broker)
Tag(s) are taken from the tags field in the message. This field must be a map type - see below
Value fields are taken from the rest of the message, and must be numeric or boolean
Measurement name can be specified as a field of the message, or hardcoded in the connector config.

Kafka Connect and Schemas

Wed, 22 Jan 2020 00:26:03 +0000

Here’s a fun one that Kafka Connect can sometimes throw out:

java.lang.ClassCastException: 
java.util.HashMap cannot be cast to org.apache.kafka.connect.data.Struct

HashMap? Struct? HUH?

Monitoring Sonos with ksqlDB, InfluxDB, and Grafana

Tue, 21 Jan 2020 22:47:35 +0000

I’m quite a fan of Sonos audio equipment but recently had some trouble with some of the devices glitching and even cutting out whilst playing. Under the covers Sonos stuff is running Linux (of course) and exposes some diagnostics through a rudimentary frontend that you can access at http://<sonos player IP>:1400/support/review:

Whilst this gives you the current state, you can’t get historical data on it. It felt like the problems were happening "all the time", but were they actually? For that, we need some cold, hard, data! Something like this, in fact:

UnsupportedClassVersionError: `` has been compiled by a more recent version of the Java Runtime

Tue, 21 Jan 2020 22:26:00 +0000

This article is just for Googlers and my future self encountering this error. Recently I was building a Docker image from the ksqlDB code base, and whilst it built successfully the ksqlDB server process in the Docker container when instantiated failed with a UnsupportedClassVersionError:

Changing the Logging Level for Kafka Connect Dynamically

Thu, 16 Jan 2020 22:50:45 +0000

Logs are magical things. They tell us what an application is doing—or not doing. They help us debug problems. As it happens, they also underpin the entire philosophy of Apache Kafka, but that’s a story for another day. Today we’re talking about logs written by Kafka Connect, and how we can change the amount of detail written.

By default, Kafka Connect will write logs at INFO and above. So when it starts up, the settings that it’s using, and any WARN or ERROR messages along the way - a missing configuration, a broken connector, and so on. If you want to peer under the covers of what’s happening, perhaps in a given connector, you’d want to see DEBUG or even TRACE messages too.

How to win [or at least not suck] at the conference abstract submission game

Thu, 16 Jan 2020 13:45:31 +0000

Just over a year ago, I put together the crudely-titled "Quick Thoughts on Not Writing a Crap Abstract" after reviewing a few dozen conference abstracts. This time around I’ve had the honour of being on a conference programme committee and with it the pleasure of reading 250+ abstracts—from which I have some more snarky words of wisdom to impart on the matter.

Remind me…how does this conference game work?

Before we really get into it, let’s recap how this whole game works, because plenty of people are new to conference speaking.

Exploring ksqlDB window start time

Thu, 09 Jan 2020 14:25:01 +0000

Prompted by a question on StackOverflow I had a bit of a dig into how windows behave in ksqlDB, specifically with regards to their start time. This article shows also how to create test data in ksqlDB and create data to be handled with a timestamp in the past.

For a general background to windowing in ksqlDB see the excellent docs.

The nice thing about recent releases of ksqlDB/KSQL is that you can create and populate streams directly with CREATE STREAM and INSERT INTO respectively. Much as I love kafkacat, being able to build a whole example within the ksqlDB CLI is very useful.

Streaming messages from RabbitMQ into Kafka with Kafka Connect

Wed, 08 Jan 2020 13:06:57 +0000

This was prompted by a question on StackOverflow to which I thought the answer would be straightforward, but turned out not to be so. And then I got a bit carried away and ended up with a nice example of how you can handle schema-less data coming from a system such as RabbitMQ and apply a schema to it.

This same pattern for ingesting bytes and applying a schema will work with other connectors such as MQTT

Analysing network behaviour with ksqlDB and MongoDB

Fri, 20 Dec 2019 17:23:40 +0000

In this post I want to build on my previous one and show another use of the Syslog data that I’m capturing. Instead of looking for SSH attacks, I’m going to analyse the behaviour of my networking components.

You can find all the code to run this on GitHub.

Getting Syslog data into Kafka

As before, let’s create ourselves a syslog connector in ksqlDB:

CREATE SOURCE CONNECTOR SOURCE_SYSLOG_UDP_01 WITH (
    'tasks.max' = '1',
    'connector.class' = 'io.confluent.connect.syslog.SyslogSourceConnector',
    'topic' = 'syslog',
    'syslog.port' = '42514',
    'syslog.listener' = 'UDP',
    'syslog.reverse.dns.remote.ip' = 'true',
    'confluent.license' = '',
    'confluent.topic.bootstrap.servers' = 'kafka:29092',
    'confluent.topic.replication.factor' = '1'
);

Detecting and Analysing SSH Attacks with ksqlDB

Wed, 18 Dec 2019 17:23:40 +0000

I’ve written previously about ingesting Syslog into Kafka and using KSQL to analyse it. I want to revisit the subject since it’s nearly two years since I wrote about it and some things have changed since then.

ksqlDB now includes the ability to define connectors from within it, which makes setting things up loads easier.

You can find the full rig to run this on GitHub.

Create and configure the Syslog connector

To start with, create a source connector:

Copy MongoDB collections from remote to local instance

Tue, 17 Dec 2019 20:23:49 +0000

This is revisiting the blog I wrote a while back, which showed using mongodump and mongorestore to copy a MongoDB database from one machine (a Unifi CloudKey) to another. This time instead of a manual lift and shift, I wanted a simple way to automate the update of the target with changes made on the source.

The source is as before, Unifi’s CloudKey, which runs MongoDB to store its data about the network - devices, access points, events, and so on.

Kafka Connect - Request timed out

Fri, 29 Nov 2019 14:37:24 +0000

A short & sweet blog post to help people Googling for this error, and me next time I encounter it.

The scenario: trying to create a connector in Kafka Connect (running in distributed mode, one worker) failed with the curl response

HTTP/1.1 500 Internal Server Error
Date: Fri, 29 Nov 2019 14:33:53 GMT
Content-Type: application/json
Content-Length: 48
Server: Jetty(9.4.18.v20190429)

{"error_code":500,"message":"Request timed out"}

Using tcpdump With Docker

Fri, 29 Nov 2019 11:17:24 +0000

I was doing some troubleshooting between two services recently and wanting to poke around to see what was happening in the REST calls between them. Normally I’d reach for tcpdump to do this but imagine my horror when I saw:

root@ksqldb-server:/# tcpdump
bash: tcpdump: command not found

Common mistakes made when configuring multiple Kafka Connect workers

Fri, 22 Nov 2019 11:33:48 +0000

Kafka Connect can be deployed in two modes: Standalone or Distributed. You can learn more about them in my Kafka Summit London 2019 talk.

I usually recommend Distributed for several reasons:

It can scale
It is fault-tolerant
It can be run on a single node sandbox or a multi-node production environment
It is the same configuration method however you run it

I usually find that Standalone is appropriate when:

You need to guarantee locality of task execution, such as picking up a log file from a folder on a specific machine
You don’t care about scale or fault-tolerance ;-)
You like re-learning how to configure something when you realise that you do care about scale or fault-tolerance X-D

Streaming data from SQL Server to Kafka to Snowflake ❄️ with Kafka Connect

Wed, 20 Nov 2019 17:59:50 +0000

Snowflake is the data warehouse built for the cloud, so let’s get all ☁️ cloudy and stream some data from Kafka running in Confluent Cloud to Snowflake!

What I’m showing also works just as well for an on-premises Kafka cluster. I’m using SQL Server as an example data source, with Debezium to capture and stream and changes from it into Kafka.

I’m assuming that you’ve signed up for Confluent Cloud and Snowflake and are the proud owner of credentials for both. I’m going to use a demo rig based on Docker to provision SQL Server and a Kafka Connect worker, but you can use your own setup if you want.

Running Dockerised Kafka Connect worker on GCP

Tue, 12 Nov 2019 14:45:43 +0000

I talk and write about Kafka and Confluent Platform a lot, and more and more of the demos that I’m building are around Confluent Cloud. This means that I don’t have to run or manage my own Kafka brokers, Zookeeper, Schema Registry, KSQL servers, etc which makes things a ton easier. Whilst there are managed connectors on Confluent Cloud (S3 etc), I need to run my own Kafka Connect worker for those connectors not yet provided. An example is the MQTT source connector that I use in this demo. Up until now I’d either run this worker locally, or manually build a cloud VM. Locally is fine, as it’s all Docker, easily spun up in a single docker-compose up -d command. I wanted something that would keep running whilst my laptop was off, but that was as close to my local build as possible—enter GCP and its functionality to run a container on a VM automagically.

You can see the full script here. The rest of this article just walks through the how and why.

Debezium & MySQL v8 : Public Key Retrieval Is Not Allowed

Wed, 23 Oct 2019 11:54:51 -0400

I started hitting problems when trying Debezium against MySQL v8. When creating the connector:

Using Kafka Connect and Debezium with Confluent Cloud

Wed, 16 Oct 2019 16:29:34 +0100

This is based on using Confluent Cloud to provide your managed Kafka and Schema Registry. All that you run yourself is the Kafka Connect worker.

Optionally, you can use this Docker Compose to run the worker and a sample MySQL database.

Skipping bad records with the Kafka Connect JDBC sink connector

Tue, 15 Oct 2019 09:58:38 +0100

The Kafka Connect framework provides generic error handling and dead-letter queue capabilities which are available for problems with [de]serialisation and Single Message Transforms. When it comes to errors that a connector may encounter doing the actual pull or put of data from the source/target system, it’s down to the connector itself to implement logic around that. For example, the Elasticsearch sink connector provides configuration (behavior.on.malformed.documents) that can be set so that a single bad record won’t halt the pipeline. Others, such as the JDBC Sink connector, don’t provide this yet. That means that if you hit this problem, you need to manually unblock it yourself. One way is to manually move the offset of the consumer on past the bad message.

TL;DR : You can use kafka-consumer-groups --reset-offsets --to-offset <x> to manually move the connector past a bad message

Kafka Connect and Elasticsearch

Mon, 07 Oct 2019 15:44:59 +0100

I use the Elastic stack for a lot of my talks and demos because it complements Kafka brilliantly. A few things have changed in recent releases and this blog is a quick note on some of the errors that you might hit and how to resolve them. It was inspired by a lot of the comments and discussion here and here.

Copying data between Kafka clusters with Kafkacat

Sun, 29 Sep 2019 10:43:45 +0200

kafkacat gives you Kafka super powers 😎

I’ve written before about kafkacat and what a great tool it is for doing lots of useful things as a developer with Kafka. I used it too in a recent demo that I built in which data needed manipulating in a way that I couldn’t easily elsewhere. Today I want share a very simple but powerful use for kafkacat as both a consumer and producer: copying data from one Kafka cluster to another. In this instance it’s getting data from Confluent Cloud down to a local cluster.

Kafka Summit GoldenGate bridge run/walk

Mon, 23 Sep 2019 10:55:23 +0100

Coming to Kafka Summit in San Francisco next week? Inspired by similar events at Oracle OpenWorld in past years, I’m proposing an unofficial run (or walk) across the GoldenGate bridge on the morning of Tuesday 1st October. We should be up and out and back in plenty of time to still attend the morning keynotes. Some people will run, some may prefer to walk, it’s open to everyone :)

Staying sane on the road as a Developer Advocate

Thu, 19 Sep 2019 23:38:42 +0100

I’ve been a full-time Developer Advocate for nearly 1.5 years now, and have learnt lots along the way. The stuff I’ve learnt about being an advocate I’ve written about elsewhere (here/here/here); today I want to write about something that’s just as important: staying sane and looking after yourself whilst on the road. This is also tangentially related to another of my favourite posts that I’ve written: Travelling for Work, with Kids at Home.

Where I'll be on the road for the remainder of 2019

Mon, 02 Sep 2019 17:36:01 +0100

I’ve had a relaxing couple of weeks off work over the summer, and came back today to realise that I’ve got a fair bit of conference and meetup travel to wrap my head around for the next few months :)

If you’re interested in where I’ll be and want to come and say hi, hear about Kafka—or just grab a coffee or beer, herewith my itinerary as it currently stands.

Reset Kafka Connect Source Connector Offsets

Thu, 15 Aug 2019 10:42:34 +0100

Kafka Connect in distributed mode uses Kafka itself to persist the offsets of any source connectors. This is a great way to do things as it means that you can easily add more workers, rebuild existing ones, etc without having to worry about where the state is persisted. I personally always recommend using distributed mode, even if just for a single worker instance - it just makes things easier, and more standard. Watch my talk online here to understand more about this. If you want to reset the offset of a source connector then you can do so by very carefully modifying the data in the Kafka topic itself.

Starting a Kafka Connect sink connector at the end of a topic

Fri, 09 Aug 2019 17:11:06 +0200

When you create a sink connector in Kafka Connect, by default it will start reading from the beginning of the topic and stream all of the existing—and new—data to the target. The setting that controls this behaviour is auto.offset.reset, and you can see its value in the worker log when the connector runs:

[2019-08-05 23:31:35,405] INFO ConsumerConfig values:
        allow.auto.create.topics = true
        auto.commit.interval.ms = 5000
        auto.offset.reset = earliest
…

Resetting a Consumer Group in Kafka

Fri, 09 Aug 2019 16:32:46 +0200

I’ve been using Replicator as a powerful way to copy data from my Kafka rig at home onto my laptop’s Kafka environment. It means that when I’m on the road I can continue to work with the same set of data and develop pipelines etc. With a VPN back home I can even keep them in sync directly if I want to.

I hit a problem the other day where Replicator was running, but I had no data in my target topics on my laptop. After a bit of head-scratching I realised that my local Kafka environment had been rebuilt (I use Docker Compose so complete rebuilds to start from scratch are easy), hence no data in the topic. But, even after restarting the Replicator Kafka Connect worker, I still had no data loaded into the empty topics. What was going on? Well Replicator acts as a consumer from the source Kafka cluster (on my home server), and so far as that Kafka cluster was concerned, Replicator had already read the messages. It thought that because even though I’d rebuilt everything on my laptop, Replicator was using the same connector name as before, and the connector name is used as the Consumer group name - which is how the source Kafka cluster keeps track of the offsets. So my "new" Kafka environment was going back to the source, which viewed it as the existing "old" one, which had already received the messages.

Migrating Alfred Clipboard to New Laptop

Wed, 07 Aug 2019 14:23:33 -0700

Alfred is one of my favourite productivity tools. One of its best features is the clipboard history, which when I moved laptops and it didn’t transfer I realised quite how much I rely on this functionality in my day-to-day work.

Whilst Alfred has the options to syncronise its preferences across machines, it seems that it doesn’t synchronise the clipboard database. To get it to work I did the following:

So how DO you make those cool diagrams? July 2019 update

Thu, 11 Jul 2019 11:12:26 +0200

I write and speak lots about Kafka, and get a fair few questions from this. The most common question is actually nothing to do with Kafka, but instead:

How do you make those cool diagrams?

I wrote about this originally last year but since then have evolved my approach. I’ve now pretty much ditched Paper, in favour of Concepts. It was recommended to me after I published the previous post.

Robin, check Concepts app. IMHO the best solution on iPad + Pencil setup. Some drawing skills are still needed, but this app is f… amazing :)
— Mariusz Gil (@mariuszgil) December 11, 2018

I still use the Apple Pencil + iPad, but Concepts is vector-based, supports layers, grids, guidelines…is generally more powerful. Concepts is definitely worth the cost (and learning curve). It doesn’t have quite the same 'hand drawn' feel to it, but it’s close enough for what I want whilst not spending my life re-drawing bitmaps for every permutation of a diagram’s lifecycle :)

Taking the Vienna-Munich sleeper train

Wed, 03 Jul 2019 07:17:12 +0200

This week I was scheduled in to a couple of meetups, in Vienna and Munich. Flying is an inevitable part of travel since I also happen to like being home seeing my family and airplanes are usually the quickest way to make this happen. I don’t particularly enjoy flying, and there’s the environmental impact of it too—so when I realised that Vienna and Munich are relatively close to each other I looked at getting the train. In the UK trains are generally a few hours, and certainly not overnight (bar a couple of exceptions), so the novelty of getting a sleeper train appealed.

Manually delete a connector from Kafka Connect

Sun, 23 Jun 2019 11:39:46 +0200

Kafka Connect has as REST API through which all config should be done, including removing connectors that have been created. Sometimes though, you might have reason to want to manually do this—and since Kafka Connect running in distributed mode uses Kafka as its persistent data store, you can achieve this by manually writing to the topic yourself.

Automatically restarting failed Kafka Connect tasks

Thu, 06 Jun 2019 17:51:44 +0100

Here’s a hacky way to automatically restart Kafka Connect connectors if they fail. Restarting automatically only makes sense if it’s a transient failure; if there’s a problem with your pipeline (e.g. bad records or a mis-configured server) then you don’t gain anything from this. You might want to check out Kafka Connect’s error handling and dead letter queues too.

Putting Kafka Connect passwords in a separate file / externalising secrets

Fri, 24 May 2019 17:30:57 +0100

Kafka Connect configuration is easy - you just write some JSON! But what if you’ve got credentials that you need to pass? Embedding those in a config file is not always such a smart idea. Fortunately with KIP-297 which was released in Apache Kafka 2.0 there is support for external secrets. It’s extendable to use your own ConfigProvider, and ships with its own for just putting credentials in a file - which I’ll show here. You can read more here.

Deleting a Connector in Kafka Connect without the REST API

Wed, 22 May 2019 10:32:10 +0100

Kafka Connect exposes a REST interface through which all config and monitoring operations can be done. You can create connectors, delete them, restart them, check their status, and so on. But, I found a situation recently in which I needed to delete a connector and couldn’t do so with the REST API. Here’s another way to do it, by amending the configuration Kafka topic that Kafka Connect in distributed mode uses to persist configuration information for connectors. Note that this is not a recommended way of working with Kafka Connect—the REST API is there for a good reason :)

A poor man's KSQL EXPLODE/UNNEST technique

Thu, 09 May 2019 10:01:50 +0100

There is an open issue for support of EXPLODE/UNNEST functionality in KSQL, and if you need it then do up-vote the issue. Here I detail a hacky, but effective, workaround for exploding arrays into multiple messages—so long as you know the upper-bound on your array.

When a Kafka Connect converter is not a _converter_

Wed, 08 May 2019 10:06:50 +0100

Kafka Connect is a API within Apache Kafka and its modular nature makes it powerful and flexible. Converters are part of the API but not always fully understood. I’ve written previously about Kafka Connect converters, and this post is just a hands-on example to show even further what they are—and are not—about.

To understand more about Kafka Connect in general, check out my talk from Kafka Summit London From Zero to Hero with Kafka Connect.

Reading Kafka Connect Offsets via the REST Proxy

Thu, 02 May 2019 10:58:27 +0100

When you run Kafka Connect in distributed mode it uses a Kafka topic to store the offset information for each connector. Because it’s just a Kafka topic, you can read that information using any consumer.

Pivoting Aggregates in Ksql

Wed, 17 Apr 2019 15:42:56 +0100

Prompted by a question on StackOverflow, the requirement is to take a series of events related to a common key and for each key output a series of aggregates derived from a changing value in the events. I’ll use the data from the question, based on ticket statuses. Each ticket can go through various stages, and the requirement was to show, per customer, how many tickets are currently at each stage.

Connecting KSQL to a Secured Schema Registry

Fri, 12 Apr 2019 12:59:33 +0100

Confluent Cloud now includes a secured Schema Registry, which you can use from external applications, including KSQL.

To configure KSQL for it you need to set:

ksql.schema.registry.url=https://<Schema Registry endpoint>
ksql.schema.registry.basic.auth.credentials.source=USER_INFO
ksql.schema.registry.basic.auth.user.info=<Schema Registry API Key>:<Schema Registry API Secret>

Exploring KSQL Stream-Stream Joins

Thu, 28 Mar 2019 14:46:24 +0000

Introduction

What can you use stream-stream joins for? Can you use them to join between a stream of orders and stream of related shipments to do useful things? What’s not supported in KSQL, where are the cracks?

Terminate All KSQL Queries

Mon, 25 Mar 2019 16:45:40 +0000

Before you can drop a stream or table that’s populated by a query in KSQL, you have to terminate any queries upon which the object is dependent. Here’s a bit of jq & xargs magic to terminate all queries that are currently running

Quick Thoughts on Not Making a Crap Slide Deck

Tue, 19 Mar 2019 10:10:34 +0000

This post is the companion to an earlier one that I wrote about conference abstracts. In the same way that the last one was inspired by reviewing a ton of abstracts and noticing a recurring pattern in my suggestions, so this one comes from reviewing a bunch of slide decks for a forthcoming conference. They all look like good talks, but in several cases these great talks are fighting to get out from underneath the deadening weight of slides.

Herewith follows my highly-opinionated, fairly-subjective, and extremely-terse advice and general suggestions for slide decks. You can also find relating ramblings in this recent post too. My friend and colleague Vik Gamov also wrote a good post on this same topic, and linked to a good video that I’d recommend you watch.

Using httpie with the Kafka REST Proxy

Fri, 08 Mar 2019 15:37:42 +0000

This shows how to use httpie with the Confluent REST Proxy.

Send data

echo '{"records":[{"value":{"foo":"bar"}}]}' | \
  http POST http://localhost:8082/topics/jsontest \
  Content-Type:application/vnd.kafka.json.v2+json Accept:application/vnd.kafka.v2+json

Preparing a New Talk

Fri, 01 Mar 2019 11:00:26 +0100

I’ve written quite a few talks over the years, but usually as a side-line to my day job. In my role as a Developer Advocate, talks are part of What I Do, and so I can dedicate more time to it. A lot of the talks I’ve done previously have evolved through numerous iterations, and with a new talk to deliver for the "Spring Season" of conferences, I thought it would be interesting to track what it took from concept to actual delivery.

Travelling for Work, with Kids at Home

Sat, 09 Feb 2019 14:13:21 +0000

I began travelling for my job when my first child was three months old. But don’t mistake correlation for causation…it wasn’t the broken nights' sleep that forced me onto the road, but an excellent job opportunity that seemed worth the risk. Nearly eight years later and I’m in a different job but still with a bunch of travel involved. How much I travel has varied. It’s tended to average around 30%, but has peaked at way more than that. I’ve worked in consultancy, business development, and as a developer advocate in that time.

Kafka Connect Change Log Level and Write Log to File

Tue, 29 Jan 2019 11:15:01 -0800

By default Kafka Connect sends its output to stdout, so you’ll see it on the console, Docker logs, or wherever. Sometimes you might want to route it to file, and you can do this by reconfiguring log4j. You can also change the configuration to get more (or less) detail in the logs by changing the log level.

Finding the log configuration file

The configuration file is called connect-log4j.properties and usually found in etc/kafka/connect-log4j.properties.

Replacing UTF8 non-breaking-space with bash/sed on the Mac

Mon, 21 Jan 2019 14:01:24 +0000

A script I’d batch-run on my Markdown files had inserted a UTF-8 non-breaking-space between Markdown heading indicator and the text, which meant that # My title actually got rendered as that, instead of an H3 title.

Looking at the file contents, I could see it wasn’t just a space between the # and the text, but a non-breaking space.

How KSQL handles case

Mon, 21 Jan 2019 12:05:48 +0000

KSQL is generally case-sensitive. Very sensitive, at times ;-)

KSQL REST API cheatsheet

Thu, 17 Jan 2019 12:12:11 +0000

Full reference is here

Confluent Schema Registry REST API cheatsheet

Thu, 17 Jan 2019 11:25:40 +0000

The Schema Registry support a REST API for finding out information about the schemas within it. Here’s a quick cheatsheat with REST calls that I often use.

What to Do When Docker on the Mac Runs Out of Space

Wed, 09 Jan 2019 10:18:20 +0000

I use Docker and Docker Compose a lot. Like, every day. It’s a fantastic way to build repeatable demos and examples, that can be torn down and spun up in a repeatable way. But…what happens when the demo that was working is spun up and then tail spins down in a blaze of flames?

Quick Thoughts on Not Writing a Crap Abstract

Wed, 19 Dec 2018 22:26:04 +0000

I’ve reviewed a bunch of abstracts in the last couple of days, here are some common suggestions I made:

No need to include your company name in the abstract text. Chances are I’ve not heard of your company, and even if I have, what does it add to my comprehension of your abstract and what you’re going to talk about? Possible exception would be the "hot" tech companies where people will see a talk just because it’s Netflix etc
I really don’t want just to read your project documentation/summary. It makes me worry your talk will be death by PowerPoint of the minutiae of something that’s only relevant in your company.
Following on from above, I want to see that there’s going to be things you’ll share that are useful for other people in a similar situation. Something that’s specific to your project, your company, doesn’t translate to mass-usefulness. Something that other people will hit, whether it’s technical or org-cultural, now that is interesting and is going to be useful
If my eyes start to glaze over reading the abstract intro, already I’m assuming that your talk will make me bored too. Read it back out loud to yourself…make sure each word justifies its place in the text. Boilerplate filler and waffle should be left on the cutting room floor.
You need to strike a balance between giving enough detail about the contents of your talk that I am convinced you have interesting things to share, but without listing every nut and bolt of detail. Too much detail and it just becomes a laundry list. You need to whet people’s appetite for the actual meal, not put them off their food.
For heaven’s sake, proof read! If you can’t be arsed to use a spell checker, then I definitely wouldn’t trust you to prepare a talk of any quality. I’ve recently started using Grammarly and it’s excellent.

Moving from Ghost to Hugo

Mon, 17 Dec 2018 23:00:21 +0000

Why?

I’ve been blogging for quite a few years now, starting on Blogger, soon onto WordPress, and then to Ghost a couple of years ago. Blogger was fairly lame, WP yucky, but I really do like Ghost. It’s simple and powerful and was perfect for my needs. My needs being, an outlet for technical content that respected formatting, worked with a markup language (Markdown), and didn’t f**k things up in the way that WP often would in its WYSIWYG handling of content.

Pull new version of multiple Docker images

Mon, 17 Dec 2018 17:44:02 +0000

Tiny little snippet this one. Given a list of images:

$ docker images|grep confluent
confluentinc/cp-enterprise-kafka                5.0.0               d0c5528d7f99        3 months ago        600MB
confluentinc/cp-kafka                           5.0.0               373a4e31e02e        3 months ago        558MB
confluentinc/cp-zookeeper                       5.0.0               3cab14034c43        3 months ago        558MB
confluentinc/cp-ksql-server                     5.0.0               691bc3c1991f        4 months ago        493MB
confluentinc/cp-ksql-cli                        5.0.0               e521f3e787d6        4 months ago        488MB
…

Now there’s a new version available, and you want to pull down all the latest ones for it:

docker images|grep "^confluentinc"|awk '{print $1}'|xargs -Ifoo docker pull foo:5.1.0

Docker Tips and Tricks with Kafka Connect, ksqlDB, and Kafka

Sat, 15 Dec 2018 22:00:55 +0000

A few years ago a colleague of mine told me about this thing called Docker, and I must admit I dismissed it as a fad…how wrong was I. Docker, and Docker Compose, are one of my key tools of the trade. With them I can build self-contained environments for tutorials, demos, conference talks etc. Tear it down, run it again, without worrying that somewhere a local config changed and will break things.

Streaming data from Oracle into Kafka

Wed, 12 Dec 2018 09:49:04 +0000

This is a short summary discussing what the options are for integrating Oracle RDBMS into Kafka, as of December 2018 (refreshed June 2020). For a more detailed background to why and how at a broader level for all databases (not just Oracle) see this blog and this talk.

What techniques & tools are there?

Franck Pachot has written up an excellent analysis of the options available here.

Tools I Use: iPad Pro

Tue, 11 Dec 2018 15:12:15 +0000

I’ve written recently about how I create the diagrams in my blog posts and talks, and from discussions around that, a couple of people were interested more broadly in how I use my iPad Pro. So, on the basis that if two people are interested maybe others are (and if no-one else is, I have a copy-and-paste answer to give to those two people) here we go.

Kit

iPad Pro 10.5" (2018)
- 256GB model
- Apple Pencil
- Apple Keyboard
- iPad wallet/protector
Matte screen protector

Background

I travel quite a lot for work, so want something with a decent battery life for stuff like:

So how DO you make those cool diagrams?

Mon, 10 Dec 2018 12:38:18 +0000

I write and speak lots about Kafka, and get a fair few questions from this. The most common question is actually nothing to do with Kafka, but instead:

How do you make those cool diagrams?

So here’s a short, and longer, answer!

Update July 2019

I’ve moved away from Paper -> read more here

tl;dr

An iOS app called Paper, from a company called FiftyThree

So, how DO you make those cool diagrams?

Disclaimer: This is a style that I have copied straight from my esteemed colleagues at Confluent, including Neha Narkhede and Ben Stopford, as well as others including Martin Kleppmann.

Get mtr working on the Mac

Sat, 08 Dec 2018 12:45:40 +0000

Install

Not sure why the brew doesn’t work as it used to, but here’s how to get it working:

brew install mtr
sudo ln /usr/local/Cellar/mtr/0.92/sbin/mtr /usr/local/bin/mtr
sudo ln /usr/local/Cellar/mtr/0.92/sbin/mtr-packet /usr/local/bin/mtr-packet

(If you don’t do the two symbolic links (ln) you’ll get mtr: command not found or mtr: Failure to start mtr-packet: Invalid argument)

Run

sudo mtr google.com

Kafka Connect CLI tricks

Mon, 03 Dec 2018 14:50:45 +0000

I do lots of work with Kafka Connect, almost entirely in Distributed mode—even just with 1 node -> makes scaling out much easier when/if needed. Because I’m using Distributed mode, I use the Kafka Connect REST API to configure and manage it. Whilst others might use GUI REST tools like Postman etc, I tend to just use the commandline. Here are some useful snippets that I use all the time.

I’m showing the commands split with a line continuation character (\) but you can of course run them on a single line. You might also choose to get fancy and set the Connect host and port as environment variables etc, but I leave that as an exercise for the reader :)

ERROR: Invalid interpolation format for "command" option in service…

Tue, 20 Nov 2018 17:47:54 +0000

Doing some funky Docker Compose stuff, including:

Flatten CDC records in KSQL

Thu, 11 Oct 2018 15:13:59 +0000

The problem - nested messages in Kafka

Data comes into Kafka in many shapes and sizes. Sometimes it’s from CDC tools, and may be nested like this:

Exploring JMX with jmxterm

Wed, 19 Sep 2018 08:11:00 +0000

Check out the jmxterm repository
Download jmxterm from https://docs.cyclopsgroup.org/jmxterm

Accessing Kafka Docker containers' JMX from host

Mon, 17 Sep 2018 15:29:48 +0000

See also docs.

To help future Googlers… with the Confluent docker images for Kafka, KSQL, Kafka Connect, etc, if you want to access JMX metrics from within, you just need to pass two environment variables: <x>_JMX_HOSTNAME and <x>_JMX_PORT, prefixed by a component name.

<x>_JMX_HOSTNAME - the hostname/IP of the JMX host machine, as accessible from the JMX Client.

This is used by the JMX client to connect back into JMX, so must be accessible from the host machine running the JMX client.

Sending multiline messages to Kafka

Tue, 04 Sep 2018 08:26:51 +0000

(SO answer repost)

You can use kafkacat to send messages to Kafka that include line breaks. To do this, use its -D operator to specify a custom message delimiter (in this example /):

kafkacat -b kafka:29092 \
        -t test_topic_01 \
        -D/ \
        -P <<EOF
this is a string message 
with a line break/this is 
another message with two 
line breaks!
EOF

Note that the delimiter must be a single byte - multi-byte chars will end up getting included in the resulting message See issue #140

Window Timestamps in KSQL / Integration with Elasticsearch

Mon, 03 Sep 2018 16:16:30 +0000

KSQL provides the ability to create windowed aggregations. For example, count the number of messages in a 1 minute window, grouped by a particular column:

CREATE TABLE RATINGS_BY_CLUB_STATUS AS \
SELECT CLUB_STATUS, COUNT(*) AS RATING_COUNT \
FROM RATINGS_WITH_CUSTOMER_DATA \
     WINDOW TUMBLING (SIZE 1 MINUTES) \
GROUP BY CLUB_STATUS;

How KSQL, and Kafka Streams, stores the window timestamp associated with an aggregate, has recently changed. See #1497 for details.

Whereas previously the Kafka message timestamp (accessible through the KSQL ROWTIME system column) stored the start of the window for which the aggregate had been calculated, this changed in July 2018 to instead be the timestamp of the latest message to update that aggregate value. This was in Apache Kafka 2.0 and Confluent Platform 5.0, and back-ported to previous versions.

Where I'm speaking in the rest of 2018

Tue, 21 Aug 2018 21:01:00 +0000

There’s lots going on in the next few months :-)

I’m particularly excited to be speaking at several notable conferences for the first time, including JavaZone, USENIX LISA, and Devoxx.

As always, if you’re nearby then hope to see you there, and let me know if you want to meet for a coffee or beer!

September

October

🇬🇧 Leeds, UK

4th Oct: The JVM Thing meetup

🇺🇸 Nashville (TN), USA

31st Oct: LISA18

November

🇩🇪 Münich, Germany

7th Nov: W-JAX

🇧🇪 Antwerp, Belgium

13th Nov: Devoxx Belgium

🇵🇱 Krakow, Poland

26th Nov: CoreDump

December

🇬🇧 Liverpool, UK

4th Dec: UKOUG TECH 18

🇩🇪 Frankfurt, Germany

10th Dec: Apache Kafka Meetup
11th Dec: IT Days

Kafka Listeners - Explained

Thu, 02 Aug 2018 19:38:00 +0000

(This was cross-posted on the Confluent.io blog)

This question comes up on StackOverflow and such places a lot, so here’s something to try and help.

tl;dr : You need to set advertised.listeners (or KAFKA_ADVERTISED_LISTENERS if you’re using Docker images) to the external address (host/IP) so that clients can correctly connect to it. Otherwise they’ll try to connect to the internal host address–and if that’s not reachable then problems ensue.

Put another way, courtesy of Spencer Ruport:

LISTENERS are what interfaces Kafka binds to. ADVERTISED_LISTENERS are how clients can connect.

Syntax highlighting code for presentation slides

Wed, 20 Jun 2018 18:32:10 +0000

So you’ve got a code sample you want to share in a presentation, but whilst it looks beautiful in your text-editor with syntax highlighting, it’s fugly in Keynote? You could screenshot it and paste the image into your slide, but you just know that you’ll want to change that code, and end up re-snapshotting it…what a PITA.

Better to have a nicely syntax-highlighted code snippet that you can paste as formatted text into Keynote and amend from there as needed. Here’s how.

Analysing Network Data with Apache Kafka, KSQL, and Elasticsearch

Sun, 17 Jun 2018 11:35:20 +0000

In this article I demonstrated how to use KSQL to filter streams of network event data. As well as filtering, KSQL can be used to easily enrich streams. In this article we’ll see how this enriched data can be used to drive analysis in Elasticsearch and Kibana—and how KSQL again came into use for building some stream processing as a result of the discovery made.

The data came from my home Ubiquiti router, and took two forms:

Compare and apply a diff / patch recursively

Thu, 07 Jun 2018 14:35:36 +0000

Hacky way to keep config files in sync when there’s a new version of some software.

Caveat : probably completely wrong, may not pick up config entries added in the new version, etc. But, works for me right here right now ;-)

So let’s say we have two folders:

confluent-4.1.0
confluent-4.1.1

Same structures, different versions. 4.1.0 was set up with our local config in ./etc, that we want to preserve. We can use diff to easily see what’s changed:

Kafka Connect and Oracle data types

Mon, 21 May 2018 08:59:00 +0000

The Kafka Connect JDBC Connector by default does not cope so well with:

NUMBER columns with no defined precision/scale. You may end up with apparent junk (bytes) in the output, or just errors.
TIMESTAMP WITH LOCAL TIME ZONE. Throws JDBC type -102 not currently supported warning in the log.

Read more about NUMBER data type in the Oracle docs.

tl;dr : How do I make it work?

There are several options:

New in Confluent Platform 4.1.1 : `numeric.mapping`

In the connector configuration, set "numeric.mapping":"best_fit"
New in Confluent Platform 4.1.1 (Doc)

Avoid the problem in the first place

Change the DDL of the source object. For example:
- refine the NUMBER ’s precision and scale
- Use a TIMESTAMP type that is supported

CAST the datatypes in the `query`

Pull from the object directly, and use query in the JDBC connector (instead of table.whitelist)—and cast the columns appropriately:

Stream-Table Joins in KSQL: Stream events must be timestamped after the Table messages

Thu, 17 May 2018 10:16:43 +0000

(preserving this StackOverflow answer for posterity and future Googlers)

tl;dr When doing a stream-table join, your table messages must already exist (and must be timestamped) before the stream messages. If you re-emit your source stream messages, after the table topic is populated, the join will succeed.

Example data

Use kafakcat to populate topics:

kafkacat -b localhost:9092 -P -t sessionDetails <<EOF
{"Media":"Foo","SessionIdTime":"2018-05-17 11:25:33 BST","SessionIdSeq":1}
{"Media":"Foo","SessionIdTime":"2018-05-17 11:26:33 BST","SessionIdSeq":2}
EOF

kafkacat -b localhost:9092 -P -t voipDetails <<EOF
{"SessionIdTime":"2018-05-17 11:25:33 BST","SessionIdSeq":1,"Details":"Bar1a"}
{"SessionIdTime":"2018-05-17 11:25:33 BST","SessionIdSeq":1,"Details":"Bar1b"}
{"SessionIdTime":"2018-05-17 11:26:33 BST","SessionIdSeq":2,"Details":"Bar2"}
EOF

Validate topic contents:

Quick 'n Easy Population of Realistic Test Data into Kafka

Thu, 10 May 2018 12:56:00 +0000

tl;dr Use curl to pull data from the Mockaroo REST endpoint, and pipe it into kafkacat, thus:

curl -s "https://api.mockaroo.com/api/d5a195e0?count=2&key=ff7856d0"| \
kafkacat -b localhost:9092 -t purchases -P

Blogging v2

Wed, 09 May 2018 00:00:00 +0000

So the last post here was in 2011…seven years later I should probably post again, just to point random Google visitors to :

My new blog : https://rmoff.net
- (Wordpress is icky; Ghost FTW!)
My employer’s blog on which I also write: http://cnfl.io/rmoff

Streaming Data from MongoDB into Kafka with Kafka Connect and Debezium

Tue, 27 Mar 2018 18:52:00 +0000

Disclaimer: I am not a MongoDB person. These steps may or may not be appropriate and proper. But they worked for me :) Feel free to post in comments if I’m doing something wrong

MongoDB config - enabling replica sets

For Debezium to be able to stream changes from MongoDB, Mongo needs to have replication configured:

Docs: Replication / Convert a Standalone to a Replica Set

Stop Mongo:

rmoff@proxmox01 ~> sudo service mongod stop

Add replica set config to /etc/mongod.conf:

Cloning Ubiquiti's MongoDB instance to a separate server

Tue, 27 Mar 2018 18:45:20 +0000

DISCLAIMER: I am not a MongoDB person (even if it is Web Scale X-D) - below instructions may work for you, they may not. Use with care!

For some work I’ve been doing I wanted to access the data in Ubiquiti’s Unifi controller which it stores in MongoDB. Because I didn’t want to risk my actual Unifi device by changing local settings to enable remote access, and also because the version of MongoDB on it is older than ideal, I wanted to clone the data elsewhere. This article shows you how.

Streaming Data from MySQL into Kafka with Kafka Connect and Debezium

Sat, 24 Mar 2018 14:58:14 +0000

Debezium is a CDC tool that can stream changes from MySQL, MongoDB, and PostgreSQL into Kafka, using Kafka Connect. In this article we’ll see how to set it up and examine the format of the data. A subsequent article will show using this realtime stream of data from a RDBMS and join it to data originating from other sources, using KSQL.

The software versions used here are:

Confluent Platform 4.0
Debezium 0.7.2
MySQL 5.7.19 with Sakila sample database installed

Install Debezium

To use it, you need the relevant JAR for the source system (e.g. MySQL), and make that JAR available to Kafka Connect. Here we’ll set it up for MySQL.

KSQL: Topic … does not conform to the requirements

Tue, 06 Mar 2018 23:08:11 +0000

io.confluent.ksql.exception.KafkaTopicException: Topic 'KSQL_NOTIFY' does not conform to the requirements Partitions:1 v 4. Replication: 1 v 1

Why? Because the topic KSQL creates to underpin a CREATE STREAM AS SELECT or CREATE TABLE AS SELECT already exists, and doesn’t match what it expects. By default it will create partitions & replicas based on the same values of the input topic.

Options:

Use a different topic, via the WITH (KAFKA_TOPIC='FOO') syntax, e.g.

 CREATE STREAM TEST WITH (KAFKA_TOPIC='FOO') AS SELECT * FROM BAR;

Tell KSQL to use values that match the existing topic, with the PARTITIONS and REPLICAS parameters. So if the existing topic only has one partition, then tell KSQL that’s what you want:

Streaming data from Kafka into Elasticsearch

Tue, 06 Mar 2018 22:21:00 +0000

This article is part of a series exploring Streaming ETL in practice. You can read about setting up the ingest of realtime events from a standard Oracle platform, and building streaming ETL using KSQL.

This post shows how we take data streaming in from an Oracle transactional system into Kafka, and simply stream it onwards into Elasticsearch. This is a common pattern, for enabling rapid search or analytics against data held in systems elsewhere.

Installing the Python Kafka library from Confluent - troubleshooting some silly errors…

Tue, 06 Mar 2018 22:18:24 +0000

System:

rmoff@proxmox01:~$ uname -a
Linux proxmox01 4.4.6-1-pve #1 SMP Thu Apr 21 11:25:40 CEST 2016 x86_64 GNU/Linux

rmoff@proxmox01:~$ head -n1 /etc/os-release
PRETTY_NAME="Debian GNU/Linux 8 (jessie)"

rmoff@proxmox01:~$ python --version
Python 2.7.9

Following:

Install librdkafka, which is a pre-req for the Python library:

wget -qO - https://packages.confluent.io/deb/4.0/archive.key | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://packages.confluent.io/deb/4.0 stable main"
sudo apt-get install librdkafka-dev python-dev

Setup virtualenv:

sudo apt-get install virtualenv
virtualenv kafka_push_notify
source ./kafka_push_notify/bin/activate.fish

Try to install confluent-kafka:

Why Do We Need Streaming ETL?

Tue, 06 Mar 2018 22:18:00 +0000

(This is an expanded version of the intro to an article I posted over on the Confluent blog. Here I get to be as verbose as I like ;))

My first job from university was building a datawarehouse for a retailer in the UK. Back then, it was writing COBOL jobs to load tables in DB2. We waited for all the shops to close and do their end of day system processing, and send their data back to the central mainframe. From there it was checked and loaded, and then reports generated on it. This was nearly twenty years ago as my greying beard will attest—and not a lot has changed in the large majority of reporting and analytics systems since then. COBOL is maybe less common, but what has remained constant is the batch-driven nature of processing. Sometimes batches are run more frequently, and get given fancy names like intra-day ETL or even micro-batching. But batch processing it is, and as such latency is built into our reporting by design. When we opt for batch processing we voluntarily inject delays into the availability of data to our end users. Much better is to build our systems around a streaming platform instead.

HOWTO: Oracle GoldenGate + Apache Kafka + Schema Registry + Swingbench

Thu, 01 Feb 2018 23:15:00 +0000

This is the detailed step-by-step if you want to recreate the process I describe in the Confluent blog here

I used Oracle’s Oracle Developer Days VM, which comes preinstalled with Oracle 12cR2. You can see the notes on how to do this here. These notes take you through installing and configuring:

Swingbench, to create a sample “Order Entry” schema and simulate events on the Oracle database
Oracle GoldenGate (OGG, forthwith) and Oracle GoldenGate for Big Data (OGG-BD, forthwith)
- I’m using Oracle GoldenGate 12.3.1 which includes the Kafka Connect handler as part of its distribution. A connector for earlier versions can be found here. Some of the syntax may differ in the configuration below - if you hit problems then check out an article that I wrote with an earlier version of the tool.
OGG extract from the Order Entry schema
Confluent Platform
KSQL
Elasticsearch

From this point, I’ll now walk through configuring OGG-BD with the Kafka Connect handler

Kafka - AdminClient - Connection to node -1 could not be established. Broker may not be available

Wed, 03 Jan 2018 11:26:00 +0000

See also Kafka Listeners - Explained

A short post to help Googlers. On a single-node sandbox Apache Kafka / Confluent Platform installation, I was getting this error from Schema Registry, Connect, etc:

WARN [AdminClient clientId=adminclient-3] Connection to node -1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

KSQL was throwing a similar error:

KSQL cannot initialize AdminCLient.

I had correctly set the machine’s hostname in my Kafka server.properties:

Installing Oracle GoldenGate for Big Data 12.3.1 with Kafka Connect and Confluent Platform

Tue, 21 Nov 2017 17:31:00 +0000

Some notes that I made on installing and configuring Oracle GoldenGate with Confluent Platform. Excuse the brevity, but hopefully useful to share!

I used the Oracle Developer Days VM for this - it’s preinstalled with Oracle 12cR2. Big Data Lite is nice but currently has an older version of GoldenGate.

sudo rpm -Uvh https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
sudo yum install -y screen htop collectl rlwrap p7zip unzip sysstat perf iotop
sudo su -
cd /etc/yum.repos.d/
wget http://download.opensuse.org/repositories/shells:fish:release:2/CentOS_7/shells:fish:release:2.repo
yum install fish

Check Oracle version etc:

Where will I be at OpenWorld / Oak Table World?

Fri, 29 Sep 2017 19:02:55 +0000

Here’s where I’ll be!

If you use Google Calendar you can click on individual entries above and select copy to my calendar - which of course you’ll want to do for all the ones I’ve marked as [SPEAKING] :-)

Here’s a list of all the Apache Kafka talks at OpenWorld and JavaOne, most of which I’ll be trying to get to.

Apache Kafka™ talks at Oracle OpenWorld, JavaOne, and Oak Table World 2017

Wed, 20 Sep 2017 15:46:00 +0000

There’s an impressive 19 sessions that cover Apache Kafka™ at Oracle OpenWorld, JavaOne, and Oak Table World this year! You can find the full list with speakers in the session catalogs for OOW, JavaOne, and Oak Table World. OTW is an awesome techie conference which is at the same time as OpenWorld, next door to Moscone. Hope to see you there!

Check out the writeup of my previous visit to OOW including useful tips here.

Oracle GoldenGate / Kafka Connect Handler troubleshooting

Tue, 12 Sep 2017 21:55:16 +0000

The Replicat was kapput:

GGSCI (localhost.localdomain) 3> info rkconnoe

REPLICAT   RKCONNOE  Last Started 2017-09-12 17:06   Status ABENDED
Checkpoint Lag       00:00:00 (updated 00:46:34 ago)
Log Read Checkpoint  File /u01/app/ogg/dirdat/oe000000
                     First Record  RBA 0

So checking the OGG error log ggserr.log showed

2017-09-12T17:06:17.572-0400  ERROR   OGG-15051  Oracle GoldenGate Delivery, rkconnoe.prm:  Java or JNI exception:
                              oracle.goldengate.util.GGException: Error detected handling operation added event.
2017-09-12T17:06:17.572-0400  ERROR   OGG-01668  Oracle GoldenGate Delivery, rkconnoe.prm:  PROCESS ABENDING.

So checking the replicat log dirrpt/RKCONNOE_info_log4j.log showed:

What is Markdown, and Why is it Awesome?

Tue, 12 Sep 2017 19:00:00 +0000

Markdown is a plain-text formatting syntax. It enables you write documents in plain text, readable by others in plain text, and optionally rendered into nicely formatted PDF, HTML, DOCX etc.

It’s used widely in software documentation, particularly open-source, because it enables richer formatting than plain-text alone, but without constraining authors or readers to a given software platform.

Platforms such as github natively support Markdown rendering - so you write your README etc in markdown, and when viewed on github it is automagically rendered - without you needing to actually do anything.

Conferences & Meetups at which I'll be speaking - 2017

Mon, 11 Sep 2017 06:45:00 +0000

I’m excited to be speaking at several conferences and meetups over the next few months. Unsurprisingly, the topic will be Apache Kafka!

If you’re at any of these, please do come and say hi :)

Apache Kafka Meetup - London

My first time talking at the London Apache Kafka Meetup - always a sold-out crowd, this will be fun!

September 20th, 19:00 : Look Ma, no Code! Building Streaming Data Pipelines with Apache Kafka
- Slides are available here

Oracle OpenWorld - San Francisco

This will be my second time at OOW - I wrote up my previous trip here

Kafka Connect - JsonDeserializer with schemas.enable requires "schema" and "payload" fields

Wed, 06 Sep 2017 12:00:25 +0000

An error that I see coming up frequently in the Kafka Connect community (e.g. mailing list, Slack group, StackOverflow) is:

JsonDeserializer with schemas.enable requires "schema" and "payload" fields and may not contain additional fields

No fields found using key and value schemas for table: foo-bar

You can see an explanation, and solution, for the issue in my StackOverflow answer here: https://stackoverflow.com/a/45940013/350613

If you’re using schemas.enable in the Connector configuration, you must have schema and payload as the root-level elements of your JSON message ( Which is pretty much verbatim what the error says 😁), like this:

Simple export/import of Data Sources in Grafana

Tue, 08 Aug 2017 19:32:00 +0000

Grafana API Reference

Export all Grafana data sources to data_sources folder

mkdir -p data_sources && curl -s "http://localhost:3000/api/datasources"  -u admin:admin|jq -c -M '.[]'|split -l 1 - data_sources/

This exports each data source to a separate JSON file in the data_sources folder.

Load data sources back in from folder

This submits every file that exists in the data_sources folder to Grafana as a new data source definition.

for i in data_sources/*; do \
	curl -X "POST" "http://localhost:3000/api/datasources" \
    -H "Content-Type: application/json" \
     --user admin:admin \
     --data-binary @$i
done

Linux - USB disk connection problems - uas: probe failed with error -12

Wed, 21 Jun 2017 06:14:45 +0000

Usually connecting external disks in Linux is easy. Plug it in, run fdisk -l or lsblk | grep disk to identify the device ID, and then mount it.

Unfortunately in this instance, plugging in my Seagate 2TB wasn’t so simple. The server is running Proxmox:

# uname -a
Linux proxmox01 4.4.6-1-pve #1 SMP Thu Apr 21 11:25:40 CEST 2016 x86_64 GNU/Linux

No device showed up on lsblk or fdisk -l. In dmesg I saw:

Configuring Kafka Connect to log REST HTTP messages to a separate file

Mon, 12 Jun 2017 15:28:15 +0000

Kafka’s Connect API is a wondrous way of easily bringing data in and out of Apache Kafka without having to write a line of code. By choosing a Connector from the many available, it’s possible to set up and end-to-end data pipeline with just a few lines of configuration. You can configure this by hand, or you can use the Confluent Control Center, for both management and monitoring:

BUT … there are times when not all goes well - perhaps your source has gone offline, or one of your targets has been misconfigured. What then? Well of course, it’s diagnostics time! And for diagnostics, you need logs. When you launch Kafka Connect it logs everything to stdout, and this output includes content from the Kafka Connect REST interface. This REST interface is for configuration and control of the connectors (status/pause/resume) - and whilst Control Center is being used on the Connect configuration screens, you’ll notice that the REST interface gets polled frequently - every couple of seconds, with a greater number of requests the more connectors you have. All of this goes into the log:

kafka.common.KafkaException: No key found on line 1

Fri, 12 May 2017 00:52:41 +0000

A very silly PEBCAK problem this one, but Google hits weren’t so helpful so here goes.

Running a console producer, specifying keys:

kafka-console-producer \
--broker-list localhost:9092 \
--topic test_topic \
--property parse.key=true \
--property key.seperator=,

Failed when I entered a key/value:

1,foo
kafka.common.KafkaException: No key found on line 1: 1,foo
        at kafka.tools.ConsoleProducer$LineMessageReader.readMessage(ConsoleProducer.scala:314)
        at kafka.tools.ConsoleProducer$.main(ConsoleProducer.scala:55)
        at kafka.tools.ConsoleProducer.main(ConsoleProducer.scala)

kafka.common.KafkaException: No key found on line … but I specified the key, didn’t I?

It would help if I could spell … key.seperator isn’t a valid property to configure. separator on the other hand, is:

Keeping Up with the Deluge

Sat, 11 Mar 2017 15:30:00 +0000

How do you try and stay current on technical affairs, given only 24 hours in a day and a job to do as well? Here’s my take on it…

One of the many things that has changed perceptibly since the beginning of this century when I started working in IT is the amount of information freely available, and being created all the time. Back then, printed books and manuals were still the primary source of definitive information about a piece of software. Remember these? I bet you still have a few of them keeping your monitor at the right height still…

Install qemu on AWS EC2 Amazon Linux

Sat, 11 Mar 2017 15:04:00 +0000

Mucking about with virtual disks, I wanted to install qemu on a AWS EC2 instance in order to use qemu-img.

Not finding it in a yum repo, I built it from scratch:

$ uname -a

Linux ip-10-0-1-238 4.4.41-36.55.amzn1.x86_64 #1 SMP Wed Jan 18 01:03:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Steps:

sudo yum install -y ghc-glib-devel ghc-glib autoconf autogen intltool libtool

wget http://download.qemu-project.org/qemu-2.8.0.tar.xz
tar xvJf qemu-2.8.0.tar.xz
cd qemu-2.8.0
./configure
make
sudo make install

I hit a few errors, recorded here for passing Googlers:

Mount VMDK/OVF/OVA on Amazon Web Services (AWS) EC2

Sat, 11 Mar 2017 14:21:00 +0000

So you’ve got a Linux VM that you want to access the contents of in EC2 - how do you do it? Let’s see how. First up, convert the VMDK to raw image file. If you’ve got a ova/ovf then just untar it first (tar -xvf my_vm.ova), from which you should get the VMDK. With that, convert it using qemu-img:

$ time qemu-img convert -f vmdk -O raw SampleAppv607p-appliance-disk1.vmdk SampleAppv607p-appliance-disk1.raw

real    16m36.740s
user    6m44.136s
sys     0m11.000s

Inspect the image file:

Little Technology Wins

Sat, 11 Mar 2017 11:00:00 +0000

Wireless Headset for VOIP With No 30-Minute Dalek Timebomb

A lot of my work is done remotely, with colleagues and customers. Five years ago I bought a Microsoft LifeChat LX-3000 which plugged into the USB port on my Mac. It did the job kinda fine, with two gripes:

it wasn’t wireless. I like to wander whilst I chat, and I didn’t like being tethered. But this in itself wasn’t a reason to ditch it
After c.30 minutes on a call, my voice would turn into a dalek. or rather, my voice wouldn’t but the audio that others heard was. This happened regardless of platform (Hangouts / Zoom / Skype / etc). I figured it must be a software or network issue. Never got to the bottom of it, until I switched to using a Snowball microphone for some proper voice recording - and any calls I happened to also make on it no longer had the Dalek problem.

So I switched, on a colleague’s recommendation, to the Logitech H600. I love it. The wireless works flawlessly, and no dalek effect. Gripes? Well there’s no pleasing some people. The audio quality is great for calls, but for music I switch back to my wireless Avantree Auditions. The Logitech headset also feels a bit plastic, which I don’t care about unless I’m back here in six months complaining that it’s broken….

Time For a Change

Fri, 10 Mar 2017 17:30:00 +0000

After 5 years at Rittman Mead, 126 blog posts, 16 conferences, four published OTN articles, an Oracle ACE award - not to mention, of course, a whole heap of interesting and challenging client work - I’ve decided that it’s time to do something different.

Later this month I’ll be joining Confluent as a Partner Technology Evangelist, helping spread the good word of Apache Kafka and the Confluent platform.

As always you can find me on Twitter @rmoff, for beer tweets, fried breakfast pics - and lots of Apache Kafka! You can also email me direct at robin@rmoff.net.

HBase crash after resuming suspended VM

Fri, 20 Jan 2017 09:36:00 +0000

I use BigDataLite for a lot of my sandboxing work. This is a OVA provided by Oracle which can be run on VirtualBox, VMWare, etc and has the Cloudera Hadoop platform (CDH) along with all of Oracle’s Big Data goodies including Big Data Discovery and Big Data Spatial and Graph (BDSG).

Something that kept tripping me up during my work with BDSG was that HBase would become unavailable. Not being an HBase expert and simply using it as a data store for my property graph data, I wrote it off as mistakes on my part. But, the issue kept reoccuring enough for me to dig into it.

Kibana Timelion - Anomaly Detection

Wed, 18 Jan 2017 19:53:10 +0000

Using the holt function in Timelion to do anomaly detection on Metricbeat data in Kibana:

Expression:

$thres=0.02, .es(index='metricbeat*',metric='max:system.cpu.user.pct').lines(1).if(eq, 0, null).holt(0.9, 0.1, 0.9, 0.5h).color(#eee).lines(10).label('Prediction'), .es(index='metricbeat*',metric='max:system.cpu.user.pct').color(#666).lines(1).label(Actual), .es(index='metricbeat*',metric='max:system.cpu.user.pct').lines(1).if(eq, 0, null).holt(0.9, 0.1, 0.9, 0.5h).subtract(.es(index='metricbeat*',metric='max:system.cpu.user.pct')).abs().if(lt, $thres, null, .es(index='metricbeat*',metric='max:system.cpu.user.pct')).points(10,3,0).color(#c66).label('Anomaly').title('max:system.cpu.user.pct / @rmoff')

References:

Streaming / Unbounded Data - Resources

Mon, 16 Jan 2017 11:10:38 +0000

(img credit)

kafka-avro-console-producer - Error registering Avro schema / io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException

Fri, 02 Dec 2016 11:35:57 +0000

By default, the kafka-avro-console-producer will assume that the schema registry is on port 8081, and happily connect to it. Unfortunately, this can lead to some weird errors if another process happens to be listening on port 8081 already!

[oracle@bigdatalite tmp]$ kafka-avro-console-producer \
>  --broker-list localhost:9092 --topic kudu_test \
>  --property value.schema='{"type":"record","name":"myrecord","fields":[{"name":"id","type":"int"},{"name":"random_field", "type": "string"}]}'

{"id": 999, "random_field": "foo"}

org.apache.kafka.common.errors.SerializationException: Error registering Avro schema: {"type":"record","name":"myrecord","fields":[{"name":"id","type":"int"},{"name":"random_field","type":"string"}]}
Caused by: io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: sun.net.www.protocol.http.HttpURLConnection$HttpInputStream@4e0ae11f; line: 1, column: 2]; error code: 50005
        at io.confluent.kafka.schemaregistry.client.rest.RestService.sendHttpRequest(RestService.java:170)
        at io.confluent.kafka.schemaregistry.client.rest.RestService.httpRequest(RestService.java:187)
        at io.confluent.kafka.schemaregistry.client.rest.RestService.registerSchema(RestService.java:238)
        at io.confluent.kafka.schemaregistry.client.rest.RestService.registerSchema(RestService.java:230)
        at io.confluent.kafka.schemaregistry.client.rest.RestService.registerSchema(RestService.java:225)
        at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.registerAndGetId(CachedSchemaRegistryClient.java:59)
        at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.register(CachedSchemaRegistryClient.java:91)
        at io.confluent.kafka.serializers.AbstractKafkaAvroSerializer.serializeImpl(AbstractKafkaAvroSerializer.java:72)
        at io.confluent.kafka.formatter.AvroMessageReader.readMessage(AvroMessageReader.java:158)
        at kafka.tools.ConsoleProducer$.main(ConsoleProducer.scala:55)
        at kafka.tools.ConsoleProducer.main(ConsoleProducer.scala)

Solution? Make sure you specify the schema URL when you launch the producer, using --property schema.registry.url=http://localhost:18081 :

Oracle GoldenGate -> Kafka Connect - "Failed to serialize Avro data"

Tue, 29 Nov 2016 22:04:38 +0000

tl;dr Make sure that key.converter.schema.registry.url and value.converter.schema.registry.url are specified, and that there are no trailing whitespaces.

I’ve been building on previous work I’ve done with Oracle GoldenGate and Kafka Connect, looking at how to have the change records from the Oracle database come through to Kafka in Avro format rather than the default JSON that the sample configuration gives.

Simply changing the Kafka Connect OGG configuration file (confluent.properties) from

value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter=org.apache.kafka.connect.json.JsonConverter

Kafka Connect - java.lang.IncompatibleClassChangeError

Thu, 24 Nov 2016 20:58:44 +0000

I hit this error running Kafka Connect HDFS connector from Confluent Platform v3.1.1 on BigDataLite 4.6:

[oracle@bigdatalite ~]$ connect-standalone /etc/schema-registry/connect-avro-standalone.properties /etc/kafka-connect-hdfs/quickstart-hdfs.properties

[...]
Exception in thread "main" java.lang.IncompatibleClassChangeError: Implementing class
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:763)

The fix was to unset the CLASSPATH first:

unset CLASSPATH

boto / S3 errors

Fri, 14 Oct 2016 08:41:30 +0000

Presented without comment, warranty, or context - other than these might help a wandering code hacker.

When using SigV4, you must specify a ‘host’ parameter

boto.s3.connection.HostRequiredError: BotoClientError: When using SigV4, you must specify a 'host' parameter.

To fix, switch

conn_s3 = boto.connect_s3()

for

conn_s3 = boto.connect_s3(host='s3.amazonaws.com')

You can see a list of endpoints here.

boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request

Make sure you’re specifying the correct hostname (see above) for the bucket’s region. Determine the bucket’s region from the S3 control panel, and then use the endpoint listed here.

OGG-15051 oracle.goldengate.util.GGException: Class not found: "kafkahandler"

Fri, 29 Jul 2016 07:47:30 +0000

Similar to the previous issue, the sample config in the docs causes another snafu:

OGG-15051  Java or JNI exception:
oracle.goldengate.util.GGException:  Class not found: "kafkahandler". kafkahandler
 	Class not found: "kafkahandler". kafkahandler

This time it’s in the kafka.props file:

gg.handler.kafkahandler.Type = kafka

Should be

gg.handler.kafkahandler.type = kafka

No capital T in Type!

(Image credit: https://unsplash.com/@vanschneider)

OGG - Class not found: "com.company.kafka.CustomProducerRecord"

Thu, 28 Jul 2016 16:34:37 +0000

In the documentation for the current release of Oracle GoldenGate for Big Data (12.2.0.1.1.011) there’s a helpful sample configuration, which isn’t so helpful …

[...]
gg.handler.kafkahandler.ProducerRecordClass = com.company.kafka.CustomProducerRecord
[...]

This value for gg.handler.kafkahandler.ProducerRecordClass will cause a failure when you start the replicat:

[...]
Class not found: "com.company.kafka.CustomProducerRecord"
[...]

If you comment this configuration item out, it’ll use the default (oracle.goldengate.handler.kafka.DefaultProducerRecord) and work swimingly!

(Image credit: https://unsplash.com/@vanschneider)

Kafka Connect JDBC - Oracle - Number of groups must be positive

Wed, 27 Jul 2016 15:23:14 +0000

There are various reasons for this error, but the one I hit was that the table name is case sensitive, and returned from Oracle by the JDBC driver in uppercase.

If you specify the tablename in your connecter config in lowercase, it won’t be matched, and this error is thrown. You can validate this by setting debug logging (edit etc/kafka/connect-log4j.properties to set log4j.rootLogger=DEBUG, stdout), and observe: (I’ve truncated some of the output for legibility)

Kafka Connect - HDFS with Hive Integration - SchemaProjectorException - Schema version required

Tue, 19 Jul 2016 14:36:52 +0000

I’ve been doing some noodling around with Confluent’s Kafka Connect recently, as part of gaining a wider understanding into Kafka. If you’re not familiar with Kafka Connect this page gives a good idea of the thinking behind it.

One issue that I hit defeated my Google-fu so I’m recording it here to hopefully help out fellow n00bs.

The pipeline that I’d set up looked like this:

Eneco’s Twitter Source streaming tweets to a Kafka topic
Confluent’s HDFS Sink to stream tweets to HDFS and define Hive table automagically over them

It worked great, but only if I didn’t enable the Hive integration part. For me the integration with Hive to automatically define schemas was one of the key interests for this platform, so I wanted to see if I could get it to work. The error I got was

Configuring UPS/apcupsd

Mon, 18 Jul 2016 07:59:51 +0000

With my new server I bought a UPS, partly just as a Good Thing, but also because I suspect a powercut fried the motherboard on a previous machine that I had, and this baby is too precious to lose ;)

The idea is that the UPS will smooth out the power supply to my server, protecting it from surges or temporarily blips in power loss. If there’s a proper power cut, the UPS is connected to my server and can initiate a graceful shutdown instead of system crash. It seems unintuitive in this day and age of laptops and iPads that you just close or switch off to “suspend” them that killing the power to a server can damage it, but when you think about it just a moment more, it’s hardly surprising.

Spark sqlContext.read.json - java.io.IOException: No input paths specified in job

Wed, 13 Jul 2016 04:50:16 +0000

Trying to use SparkSQL to read a JSON file, from either pyspark or spark-shell, I got this error:

java.io.IOException: No input paths specified in job

scala> sqlContext.read.json("/u02/custom/twitter/twitter.json")
java.io.IOException: No input paths specified in job
        at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:202)

Despite the reference articles that I found using this local path syntax (/u02/custom/twitter/twitter.json), it turned out that I needed to prefix it with file://:

scala> sqlContext.read.json("file:///u02/custom/twitter/twitter.json")
res3: org.apache.spark.sql.DataFrame = [@timestamp: string, @version: string, contributors: string, coordinates: string, created_at: string, entities: struct<hashtags:array<struct<indices:array<bigint>,text:string>>,media:array<struct<display_url:string,expanded_url:string,id:bigint,id_str:string,indices:array<bigint>,media_url:string,media_url_https:string,sizes:struct<large:struct<h:bigint,resize:string,w:bigint>,medium:struct<h:bigint,resize:string,w:bigint>,small:struct<h:bigint,resize:string,w:bigint>,thumb:struct<h:bigint,resize:string,w:bigint>>,source_status_id:bigint,source_status_id_str:string,source_user_id:bigint,source_user_id_str:string,type:string,url:string>>,symbols:array<struct<indices:array<bigint>,text:string>>,urls:array<struct<display_url:string,expanded_url:string...
scala>

An alternative to file:// is hdfs://, assuming you have some data residing there too:

Proxmox 4 Containers - ssh - ssh_exchange_identification: read: Connection reset by peer

Tue, 05 Jul 2016 15:20:37 +0000

TL;DR When defining networking on Proxmox 4 LXC containers, use an appropriate CIDR suffix (e.g. 24) - don’t use 32!

On my Proxmox 4 server I’m running a whole load of lovely LXC containers. Unfortunately, I had trouble connecting to them. From a client machine, I got the error

ssh_exchange_identification: read: Connection reset by peer

On the server I was connecting to (which I could get a console for through the Proxmox GUI, or a session on using pct enter from the Proxmox host) I ran a SSHD process with debug to see what was happening:

Reset Hue password

Tue, 05 Jul 2016 13:27:06 +0000

(Ref)

The bit that caught me out was this kept failing with

Error: Password not present

and a Python stack trace that ended with

subprocess.CalledProcessError: Command '/var/run/cloudera-scm-agent/process/78-hue-HUE_SERVER/altscript.sh sec-1-secret_key' returned non-zero exit status 1

The answer (it seems) is to ensure that HUE_SECRET_KEY is set (to any value!)

Launch shell:

export HUE_SECRET_KEY=foobar
/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/build/env/bin/hue shell

Reset password for hue, activate account and make it superuser

from django.contrib.auth.models import User
user = User.objects.get(username='hue')
user.is_active=True
user.save()
user.is_superuser=True
user.save()
user.set_password('hue')
user.save()

Apache Drill - conflicting jar problem - "No current connection"

Mon, 20 Jun 2016 19:04:18 +0000

Vanilla download of Apache Drill 1.6, attempting to follow the Followed the Drill in 10 Minutes tutorial - but kept just getting the error No current connection. Here’s an example:

[oracle@bigdatalite apache-drill-1.6.0]$ ./bin/drill-embedded
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0
com.fasterxml.jackson.databind.JavaType.isReferenceType()Z
apache drill 1.6.0
"the only truly happy people are children, the creative minority and drill users"
0: jdbc:drill:zk=local> SELECT version FROM sys.version;
No current connection
0: jdbc:drill:zk=local>

Whether SELECT version FROM sys.version; or any other command - same result - No current connection. Trying to run Drill in distributed mode also failed, with a class error

ClassNotFoundException with MongoDB-Hadoop in Hive

Wed, 15 Jun 2016 17:58:19 +0000

I wasted literally two hours on this one, so putting down a note to hopefully help future Googlers.

Symptom

Here’s all the various errors that I got in the hive-server2.log during my attempts to get a CREATE EXTERNABLE TABLE to work against a MongoDB table in Hive:

Caused by: java.lang.ClassNotFoundException: com.mongodb.hadoop.io.BSONWritable
Caused by: java.lang.ClassNotFoundException: com.mongodb.util.JSON
Caused by: java.lang.ClassNotFoundException: org.bson.conversions.Bson
Caused by: java.lang.ClassNotFoundException: org.bson.io.OutputBuffer

Whilst Hive would throw errors along the lines of:

Erroneous SwapFree on LXC causes problems with CDH install

Wed, 15 Jun 2016 17:52:00 +0000

Installing CDH 5.7 on Linux Containers (LXC) hosted on Proxmox 4. Everything was going well until Cluster Setup, and which point it failed on Start YARN (MR2 included)

Completed only 0/1 steps. First failure: Failed to execute command Start on service YARN (MR2 Included)

Log /var/log/hadoop-yarn/hadoop-cmf-yarn-NODEMANAGER-cdh57-01-node-02.moffatt.me.log.out showed:

org.apache.hadoop.service.AbstractService: Service containers-monitor failed in state INITED; cause: java.lang.NumberFormatException: For input string: "18446744073709550364"
java.lang.NumberFormatException: For input string: "18446744073709550364"

Looking down the stack trace, this came from org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin.readProcMemInfoFile, which the source code shows is reading /proc/meminfo. Looking at this file on each node showed:

Reviving a bricked EdgeRouter Lite (ERL) from a Mac

Wed, 08 Jun 2016 15:58:30 +0000

I’ve got an EdgeRouter LITE (ERL) which I used as my home router until a powercut fried it a while ago (looks like I’m not the only one to have this issue). The symptoms were it powering on but not giving any DHCP addresses, or after a factory reset responding on the default IP of 192.168.1.1. It was a real shame, because it had been a great bit of kit up until then. I am a complete hack when it comes to networking, and it struck the balance right between letting me do what I needed to do, without overwhelming me with complexity. I’d replaced it with a SonicWall TZ105 but having utterly failed to get the latter to permit OpenVPN traffic (so I can access my home server when on the road), which I had done with no problem on the ERL I thought I’d try and resurrect the ERL using the instructions here.

Running a Docker Container on Proxmox for BitTorrent Sync

Tue, 07 Jun 2016 21:43:26 +0000

(Previously, previously, previously)

Since Proxmox 4 has a recent Linux kernel and mainline one at that, it means that Docker can be run on it. I’ve yet to really dig into Docker and work out when it makes sense in place of Linux Containers (LXC), so this is going to be a learning experience for me.

To install Docker, add Backports repo to apt:

root@proxmox01:~# cat /etc/apt/sources.list.d/backports.list
deb http://ftp.debian.org/debian jessie-backports main

And then install:

Importing VMWare and VirtualBox VMs to Proxmox

Tue, 07 Jun 2016 21:14:26 +0000

(Previously, previously)

I’ve got a bunch of existing VirtualBox and VMWare VMs that I want to run on Proxmox. Eventually I’ll migrate them to containers, but for the time being run them as “fat” VMs using Proxmox’s KVM virtualisation. After copying the OVA files that I had to the server, I uncompressed them:

root@proxmox01:/data04/vms/bdl44-biwa# cd ../bdl44
root@proxmox01:/data04/vms/bdl44# ll
total 27249328
-rw------- 1 root root 27903306752 Jun  1 10:14 BigDataLite440.ova
root@proxmox01:/data04/vms/bdl44# tar -xf BigDataLite440.ova
root@proxmox01:/data04/vms/bdl44# ll
total 54498668
-rw------- 1 root root  7300486656 Feb 18 21:25 BigDataLite440-disk1.vmdk
-rw------- 1 root root  1261044224 Feb 18 21:26 BigDataLite440-disk2.vmdk
-rw------- 1 root root 19295202816 Feb 18 21:48 BigDataLite440-disk3.vmdk
-rw------- 1 root root    46550528 Feb 18 21:48 BigDataLite440-disk4.vmdk
-rw------- 1 root root 27903306752 Jun  1 10:14 BigDataLite440.ova
-rw------- 1 root root       19619 Feb 18 21:15 BigDataLite440.ovf

and then converted each disk image to qcow2 format: (You can read more about how and why here and here).

Commissioning my Proxmox Server - OS and filesystems

Tue, 07 Jun 2016 21:03:22 +0000

(Previously)

With my server in place, I ran a memtest on it … which with 128G took a while ;)

And then installed Proxmox 4, using a bootable USB that I’d created on my Mac from the ISO downloaded from Proxmox’s website. To create the bootable USB, create the img file:

hdiutil convert -format UDRW -o target.img source.iso

and then burn it to USB:

sudo dd if=target.img of=/dev/rdiskN bs=1m

Replace N with the correct device based on diskutil list output. Don’t get it wrong, else you’ll properly knacker your machine :D

A New Arrival

Tue, 07 Jun 2016 20:43:20 +0000

After a long and painful delivery, I’m delighted to announce the arrival of a new addition to my household … :

This custom-build from Scan 3XS is sat in my study quietly humming away. I’m going to use it for hosting VMs for R&D on OBIEE, Big Data Lite, Elastic, InfluxDB, Kafka, etc. I’ll blog various installations that I’ve done on it as a reference for myself, and anyone else interested. Which I guess means, myself ;)

New version of BigDataLite VM from Oracle

Mon, 06 Jun 2016 22:28:25 +0000

Oracle’s excellent Big Data Lite VM has been updated, to version 4.5.

Download it here

Changes:

CDH 5.5 -> 5.7
Big Data Spatial and Graph 1.1 -> 1.2
Big Data Discovery 1.1 -> 1.2
Oracle Big Data Connectors 4.4 -> 4.5
Oracle NoSQL 3.5 -> 4.0
GoldenGate 12.2.0.1 -> 12.2.0.1.1

OBIEE 12c blog posts

Wed, 01 Jun 2016 22:30:14 +0000

I’ve been spending some interesting hours digging into OBIEE 12c recently, with some interesting blog posts to show for it. Some of it is just curiosities discovered along the way, but the real meaty stuff is the in the RESTful APIs - lots of potential here for cool integrations I think…

(Photo credit: https://unsplash.com/@jluebke)

Presentation Services Logsources in OBIEE 12c

Wed, 01 Jun 2016 11:03:00 +0000

Presentation Services can provide some very detailed logs, useful for troubleshooting, performance tracing, and general poking around. See here for details.

There’s no bi-init.sh in 12c, so need to set up the LD_LIBRARY_PATH ourselves:

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/app/oracle/biee/bi/bifoundation/web/bin/:/app/oracle/biee/bi/lib/:/app/oracle/biee/lib/:/app/oracle/biee/bi/bifoundation/odbc/lib/

Run sawserver with flag to list all log sources

/app/oracle/biee/bi/bifoundation/web/bin/sawserver -logsources > saw_logsources_12.2.1.txt

Full list: https://gist.github.com/rmoff/e3be9009da6130839c71181cb58509a0

Lifting the Lid on OBIEE 12c Web Services - Part 2

Sat, 28 May 2016 20:30:00 +0000

In OBIEE 12c data-model-cmd is a wrapper for some java code which ultimately calls an internal RESTful web service in OBIEE 12c, bi-lcm. We saw in the previous post how these internal web services can be opened up slightly, and we’re going to do the same again here. Which means, time for the same caveat:

None of these Web Services are documented, and they should therefore be assumed to be completely unsupported by Oracle. This article is purely for geek interest. Using undocumented APIs leaves you at risk of the API changing at any time.

Dynamic Naming of OBIEE 12c Service Instance Exports

Fri, 27 May 2016 09:13:00 +0000

exportServiceInstance will export the RPD, Presentation Catalog, and Security model (application roles & policies etc – but not WLS LDAP) into a single .bar file, from which they can be imported to another environment, or restored to the same one at a later date (e.g. for backup/restore).

To run exportServiceInstance you need to launch WLST first. The following demonstrates how to call it, and embeds the current timestamp & machine details in the backup (useful info, and also makes the backup name unique each time).

OBIEE 12c - "Add Data Source" in Answers

Fri, 27 May 2016 08:44:24 +0000

So this had me scratching my head for a good hour today. Comparing SampleApp v511 against a vanilla OBIEE 12c install I’d done, one had “Add Data Source” as an option in Answers, the other didn’t. The strange thing was that the option wasn’t there in SampleApp – and usually that has all the bells and whistles enabled.

After checking and re-checking the Manage Privileges option, and even the Application Policy grants, and the manual, I hit MoS - and turned up Doc ID 2093886.1.

York Fry Ups

Tue, 24 May 2016 21:45:09 +0000

I had the pleasure of not one but two fry-ups in York, UK last weekend.

The first was courtesy of Bill’s Restaurant

Overall, pretty good, and I’ve had much worse. All the ingredients seemed decent. The black pudding was overcooked and almost biscuit-like, but that’s my only serious grumble. The bacon was cooked well. That black pudding, beans and the mashed/friend potato thing were each extra charges annoyed me. Particularly with a hangover, I just want to be able to order a full english, without playing Mastermind to work out what’s in or not.

Lifting the Lid on OBIEE 12c Web Services - Part 1

Tue, 24 May 2016 21:15:00 +0000

Architecturally, OBIEE 12c is - on the surface - pretty similar to OBIEE 11g. Sure, we’ve lost OPMN in favour of Node Manager, but all the old favourites are there - WebLogic Servers, BI Server (nqsserver / OBIS), Presentation Services (sawserver / OBIPS), and so on.

But, scratch beneath the surface, or have a gander at slide decks such as this one from BIWA this year, and you realise that change is afoot. Whilst the OBIEE core is still built around proprietary ‘black box’ protocols (SAW from analytics to sawserver on port 9710, NQS ODBC from sawserver to nqsserver, cluster management on 9706 to nqsclustercontroller), there are now REST-based web services springing up (in addition to the existing SOAP services that have been there since at least 10g). Whilst the REST services are there under the covers, they are not documented nor user-servicable, but they are there. But let me re-iterate:

Kibana Timelion - Series Calculations - Difference from One Week Ago

Mon, 23 May 2016 09:46:28 +0000

I wrote recently about Kibana’s excellent Timelion feature, which brings time-series visualisations to Kibana. In the comments Ben Huang asked:

do you know how to show whats the difference between this Friday and last Friday by Timelion?

So I thought I’d answer properly here.

Timelion includes mathematical functions including add and subtract, as well as the ability to show data offset by an amount of time. So to answer Ben’s query, we combine the two.

OBIEE 12c hangs at startup - Starting AdminServer ...

Fri, 20 May 2016 14:22:21 +0000

Running the OBIEE 12c startup on Windows:

C:\app\oracle\fmw\user_projects\domains\bi\bitools\bin\start.cmd

Just hangs at:

Starting AdminServer ...

No CPU being consumed, very odd. But then … looking at DOMAIN_HOME\servers\AdminServer\logs\AdminServer.out shows the last log entry was:

Enter username to boot WebLogic server:

And that’s bad news, cos that’s an interactive prompt, but not echo’d to the console output of the startup command, and there’s no way to interact with it.

The start.cmd was being called by adding it to the Startup folder (C:\ProgramData\Microsoft\Windows\Start Menu\Programs\StartUp), and I guess it was something about this that stopped the prompt coming back to the console, because when I ran it manually from the command prompt, I got this:

oracle.bi.bar.exceptions.UnSupportedBarException: The Bar file provided as input is not supported in this BI Platfrom release.

Thu, 19 May 2016 10:06:03 +0000

Another quick note on OBIEE 12c, this time on the importServiceInstance command. If you run it with a BAR file that doesn’t exist, it’ll fail (obviously), but the error at the end of the stack trace is slightly confusing:

oracle.bi.bar.exceptions.UnSupportedBarException: 
The Bar file provided as input is not supported in this BI Platfrom release.

Scrolling back up the stack trace does show the error message:

SEVERE: Failed in reading bar file. [...]
java.io.FileNotFoundException: [...] 
(The system cannot find the file specified)

So … RTEM (Read the Fantastic Error Message) in full, don’t just skim to the end…

OBIEE Baseline Validation Tool - Parameter 'directory' is not a directory

Wed, 18 May 2016 15:35:46 +0000

Interesting quirk in running Baseline Validation Tool for OBIEE here. If you invoke obibvt from the bin folder, it errors with Parameter ‘directory’ is not a directory

PS C:\OracleBI-BVT> cd bin
PS C:\OracleBI-BVT\bin> .\obibvt -config C:\OracleBI-BVT\bin\bvt-config.xml -deployment current
 INFO: Result folder: Results\current
Throwable: Parameter 'directory' is not a directory
Thread[main,5,main]
SEVERE: Unhandled Exception
SEVERE: java.lang.IllegalArgumentException: Parameter 'directory' is not a directory
       at org.apache.commons.io.FileUtils.validateListFilesParameters(FileUtils.java:545)
       at org.apache.commons.io.FileUtils.listFiles(FileUtils.java:521)
       at org.apache.commons.io.FileUtils.listFiles(FileUtils.java:691)
       at com.oracle.biee.bvt.UpgradeTool.loadPlugins(UpgradeTool.java:537)
       at com.oracle.biee.bvt.UpgradeTool.runPluginTests(UpgradeTool.java:644)
       at com.oracle.biee.bvt.UpgradeTool.run(UpgradeTool.java:812)
       at com.oracle.biee.bvt.UpgradeTool.main(UpgradeTool.java:999)

PS C:\OracleBI-BVT\bin>

Solution? Run the exact same command, but from the folder above:

Monitoring Logstash Ingest Rates with Elasticsearch, Kibana, and Timelion

Fri, 13 May 2016 05:45:19 +0000

Yesterday I wrote about Monitoring Logstash Ingest Rates with InfluxDB and Grafana, in which InfluxDB provided the data store for the ingest rate data, and Grafana the frontend.

Mark Walkom reminded me on twitter that the next release of Logstash will add more functionality in this area - and that it’ll integrate back into the Elastic stack:

@rmoff nice, LS 5.0 will have APIs exposing metrics too. they’ll be integrated back into Marvel/Monitoring! :)

Monitoring Logstash Ingest Rates with InfluxDB and Grafana

Thu, 12 May 2016 20:56:38 +0000

In this article I’m going to show you how to easily monitor the rate at which Logstash is ingesting data, as well as in future articles the rate at which Elasticsearch is indexing it. It’s a nice little touch to add to any project involving Logstash, and it’s easy to do.

Logstash is powerful tool for data ingest, processing, and distribution. It originated as simply the pipe to slurp at log files and put them into Elasticsearch, but has evolved into a whole bunch more. With connectors to JDBC and Kafka, as well as many other input and output options (not to mention the filtering possibilities), it really is a great bit of software to use. I’ve used it over the years with OBIEE, as well as more recently to pull data from Oracle, and even IRC. Another great set of tools is InfluxDB and Grafana, which for me really round off the standalone Elastic platform (previously known as ELK - Elasticsearch, Logstash, and Kibana). What InfluxDB and Grafana give is a powerful dedicated time series database and flexible time series-based dashboarding tool respectively. A topic for another day is the Elasticsearch vs InfluxDB overlap, and Kibana vs Grafana - but for now, just take it as read that it’s horses for course, right tool for the right job, etc.

Collection of Articles on How to Write a Good Conference Abstract

Thu, 05 May 2016 09:57:00 +0000

Here’s a collection of useful articles that I’ve found over the years that give good advice on writing a good abstract, mistakes to avoid, etc:

(post photo courtesy of Calum MacAulay on https://unsplash.com)

Using R to Denormalise Data for Analysis in Kibana

Sun, 24 Apr 2016 12:22:12 +0000

Kibana is a tool from Elastic that makes analysis of data held in Elasticsearch really easy and very powerful. Because Elasticsearch has very loose schema that can evolve on demand it makes it very quick to get up and running with some cool visualisations and analysis on any set of data. I demonstrated this in a blog post last year, taking a CSV file and loading it into Elasticsearch via Logstash.

This is all great, but the one real sticking point with analytics in Elasticsearch/Kibana is that it needs the data to be denormalised. That is, you can’t give it a bunch of sources of data and it perform the joins for you in Kibana - it just doesn’t work like that. If you’re using Elasticsearch alone for analytics, maybe with a bespoke application, there are ways of approaching it, but not through Kibana. Now, depending on where the data is coming from, this may not be a problem. For example, if you use the JDBC Logstash input to pull from an RDBMS source you can specify a complex SQL query going across multiple tables, so that the data when it hits Elasticsearch is nice and denormalised and ready for fun in Kibana. But, source data doesn’t always come this way, and it’s useful to have a way to work with the data still when it is like this.

OBIEE security patches, and FINAL 11.1.1.7 patchset release

Mon, 18 Apr 2016 15:36:00 +0000

Two vulns for OBIEE in the latest critical patch update (CPU): http://www.oracle.com/technetwork/security-advisory/cpuapr2016v3-2985753.html?elq_mid=45463&sh=91225181314122121267715271910&cmid=WWMK10067711MPP001C140

Patches is bundle patch .160419:

Note that April 2016 is the last regular patchset for 11.1.1.7, ref: https://support.oracle.com/epmos/faces/DocumentDisplay?id=2102148.1#mozTocId410847. If you’re still on it, or earlier, time to upgrade!

(Photo credit: https://unsplash.com/@jenlittlebirdie)

Streaming Data through Oracle GoldenGate to Elasticsearch

Thu, 14 Apr 2016 22:51:43 +0000

Recently added to the oracledi project over at java.net is an adaptor enabling Oracle GoldenGate (OGG) to send data to Elasticsearch. This adds a powerful alternative to [micro-]batch extract via JDBC from Oracle to Elasticsearch, which I wrote about recently over at the Elastic blog.

Elasticsearch is a ‘document store’ widely used for both search and analytics. It’s something I’ve written a lot about (here and here for archives), as well as spoken about - preaching the good word, as it were, since the Elastic stack as a whole is very very good at what it does and a pleasure to work with. So, being able to combine that with my “day job” focus of Oracle is fun. Let’s get started!

Decoupling the Data Pipeline with Kafka - A (Very) Simple Real Life Example

Tue, 12 Apr 2016 21:50:46 +0000

I’ve recently been playing around with the ELK stack (now officially known as the Elastic stack) collecting data from an IRC channel with Elastic’s Logstash, storing it in Elasticsearch and analysing it with Kibana. But, this isn’t an “ELK” post - this is a Kafka post! ELK is just some example data manipulation tooling that helps demonstrate the principles.

As I wrote about last year, Apache Kafka provides a handy way to build flexible “pipelines”. Today I’m writing up a short real-world example of this in practice. There are three elements to the flexibility that I really want to highlight:

Food Pr0n 02 - Devon & Dorset

Mon, 11 Apr 2016 19:00:56 +0000

On a family holiday in South Devon last week I had some good food experiences. Top of the pile was a Lardy Cake, a delicacy new to me but which I’ll be sure to be searching out again. It reminded me of an Eccles cake, but bigger and lardier!

I got mine from Vinnicombes on the High Street in Sidmouth.

Just down the coast from Sidmouth is a village called Beer. I didn’t have any food of note there, but it’s called Beer which is certainly worth recording :)

Experiments with Kibana Timelion

Tue, 29 Mar 2016 21:07:00 +0000

Timelion was released in November 2015 and with the 4.4.2 release of Kibana is available as a native visualisation once installed. It adds some powerful capabilities to Kibana as an timeseries analysis tool, using its own data manipulation language.

Installing Timelion is a piece of cake:

./bin/kibana plugin -i kibana/timelion

After restarting Kibana, you’ll see it as an option from the application picker

There’s a bit of a learning curve with Timelion, but it’s worth it. The blog gives some basics, and the built-in help is really good too:

Connecting to OBIEE via JDBC - with jisql

Mon, 28 Mar 2016 21:01:00 +0000

OBIEE supports JDBC as a connection protocol, using the driver available on all installations of OBIEE, bijdbc.jar. This makes connecting to OBIEE from custom or third-party applications very easy. Once connected, you issue “Logical SQL” against the “tables” of the Presentation Layer. An example of logical SQL is:

SELECT "Time"."T05 Per Name Year" saw_0 FROM "A - Sample Sales"

To find more Logical SQL simply inspect your nqquery.log (obis-query.log in 12c), or Usage Tracking.

My latest IRC client : Kibana

Thu, 24 Mar 2016 21:38:02 +0000

OK, maybe that’s not entirely true. But my read-only client, certainly.

I was perusing the Logstash input plugins recently when I noticed that there was one for IRC. Being a fan of IRC and a regular on the #obihackers channel, I thought this could be fun and yet another great example of how easy the Elastic stack is to work with.

Installation is a piece of cake:

wget https://download.elasticsearch.org/elasticsearch/release/org/elasticsearch/distribution/zip/elasticsearch/2.2.1/elasticsearch-2.2.1.zip
wget https://download.elastic.co/logstash/logstash/logstash-2.2.2.zip
wget https://download.elastic.co/kibana/kibana/kibana-4.4.2-linux-x64.tar.gz
unzip \*.zip
tar -xf kibana-4.4.2-linux-x64.tar.gz
sudo mv elasticsearch-2.2.1 logstash-2.2.2 kibana-4.4.2-linux-x64 /opt

(you’ll also need Oracle JDK installed if not already, here’s a handy way to get it from the CLI).

Food Pr0n - 01

Sat, 19 Mar 2016 21:18:00 +0000

One of the perks of my job is that I get to travel to some pretty nice places (hi, San Francisco, Bergen, Åland Islands), and get to eat some pretty good food too. If you’re looking for some techie content, then move along and go and read about Kafka, but if you enjoy food pr0n then stay put.

I was working for a client in the centre of Manchester this week, staying as usual at a Premier Inn. I’m a big fan of my fried breakfasts, and whilst never anything to write home about (or indeed write a blog about) Premier Inn breakfasts are generally fine for what they are – but this one was a real disappointment. Even with the excitement of seeing bubble and squeak, it was a real let down; luke warm, and the sausages in particular tasted really artificially flavoured.

OBIEE 11.1.1.9 installation - JPS-06514: Opening of file based keystore failed

Fri, 18 Mar 2016 18:04:07 +0000

I got this lovely failure during a fresh install of OBIEE 11.1.1.9. I emphasise that it was during the install because there’s other causes for this error on an existing system to do with corrupted credential stores etc – not the case here.

The install had copied in the binaries and was in the process of building the domain. During the early stages of this where it starts configuring and restarting the AdminServer it failed, with the AdminServer.log showing the following: (I’ve extracted the salient errors from the log)

Fun and Games with Oracle GoldenGate, Kafka, and Logstash on BigDataLite 4.4

Wed, 16 Mar 2016 22:01:00 +0000

The Oracle by Example (ObE) here demonstrating how to use Goldengate to replicate transactions big data targets such as HDFS is written for the BigDataLite 4.2.1, and for me didn’t work on the current latest version, 4.4.0.

The OBE (and similar Hands On Lab PDF) assume the presence of pmov.prm and pmov.properties in /u01/ogg/dirprm/. On BDL 4.4 there’s only the extract to from Oracle configuration, emov.

Fortunately it’s still possible to run this setup out of the box in BDL 4.4, with bells on because it includes Kafka too. And, who doesn’t like a bit of Kafka nowadays?

Presentation Slides… bye-bye Slideshare, hello Speakerdeck

Wed, 09 Mar 2016 09:43:30 +0000

I’ve always defaulted to Slideshare for hosting slides from presentations that I’ve given, but it’s become more and more crap-infested. The UI is messy, and the UX sucks - for example, I want to download a slide deck, I most definitely 100% am not interested in “clipping” it…even if you ask me every. damn. time:

Looking around it seems the other popular option is Speakerdeck. The UI is clean and simple, and I as both a user and uploader I feel like I’m there to read and share slides rather than be monetised as an eyeball on the site.

obihackers IRC channel

Thu, 03 Mar 2016 22:55:37 +0000

`#obihackers`

There’s a #obihackers IRC channel on freenode, where a dozen or so of us have hung out for several years now. Chat is usually OBIEE, Oracle, ODI, and general geek out.

Bear in mind this is the equivalent of us hanging out in a bar; if you wanna shoot the shit with a geeky question about OBIEE go ahead, but if you’ve come to get help with your homework without even buying a round, you’ll probably get short shrift… ;-)

Streaming data to InfluxDB from any bash command

Sat, 27 Feb 2016 21:05:00 +0000

InfluxDB is a great time series database, that’s recently been rebranded as part of the “TICK” stack, including data collectors, visualisation, and ETL/Alerting. I’ve yet to really look at the other components, but InfluxDB alone works just great with my favourite visualisation/analysis tool for time series metrics, Grafana.

Getting data into InfluxDB is easy, with many tools supporting the native InfluxDB line input protocol, and those that don’t often supporting the carbon protocol (from Graphite), which InfluxDB also supports (along with others). So for collecting broad ranges of OS stats, for example, collectl via carbon and nmon via nmon2influxdb are both viable options.

What makes a good Full English?

Fri, 26 Feb 2016 18:02:31 +0000

Thanks to the power of twitter, I can look back on all the many and varied Full English breakfasts that I’ve (mostly) enjoyed:

👉🏻 https://twitter.com/search?q=rmoff%20%23fullenglish&src=typd

What makes a good Full English?

Good ingredients, cooked well. Nothing worse than a limp pink sausage, as it were.
Sausage, standard pork or Cumberland at most. Definitely no daft apricot and guava bean with a hint of foie gras nonsense. Must be cooked right well, crispy skin, almost burnt.
Bacon, starting to crisp (but not deep fried and crisp-like as the Americans do it - it should bend if you pick it up)
Black Pudding, seriously. Grim idea when you think about it (blood sausage) but so very tasty, and binds the other components together
Fried eggs with runny yolks
Scrambled eggs are making an appearance more recently too, after 30 years of not really caring for them. The dodgy “scrambled egg” that you get in some hotels I think straight from a packet mix are crap though and best avoided.
Hash browns. Controversial, and often a bit greasy - but I’d never pass one by
Fried slice - sadly all too often missing. I’ll forego hash browns if there’s a fried slice
Tomatoes, but only if there’s more tomato than stalk as often happens
Mushrooms, thicky chopped so there’s something to it - and certainly not the wussy sliced things that you get sitting in a pool of black greasy water at some fryup buffets
Baked beans
Good white toast
HP Sauce - although I’ve realised recently that a really good fryup doesn’t actually need any sauce. But if it’s subpar and/or cheap ingrediants – gotta have HP sauce (and definitely not that awful “brown sauce” that you get in sachets and tastes grim).

Visualising OBIEE DMS Metrics over the years

Fri, 26 Feb 2016 17:54:54 +0000

It struck me today when I was writing my most recent blog over at Rittman Mead that I’ve been playing with visualising OBIEE metrics for years now.

Back in 2009 I wrote about using something called JManage to pull metrics out of OBIEE 10g via JMX:
Still with OBIEE 10g in 2011, I was using rrdtool and some horrible-looking tcl hacking to get the metrics out through jmx :
2014 brought with it DMS and my first forays with Graphite for storing & visualising data:

Blogging

Mon, 28 Nov 2011 00:00:00 +0000

I will now be blogging mostly over at the venerable blog of my employer, Rittman Mead.

You can see my first posting here: Web Services in BI Publisher 11g.

Don’t entirely exclude this rnm1978 blog from your feeds, as I may still post more esoteric and random tidbits here.

Instrumenting OBIEE - the final chapter

Mon, 10 Oct 2011 00:00:00 +0000

This article has been superseded by a newer version: Instrumenting OBIEE Database Connections For Improved Performance Diagnostics

(Previously on this blog: 1, 2, 3…)

Summary

Instrument your code. Stop guessing. Make your DBA happy. Make your life as a BI Admin easier.

The Problem

OBIEE will typically connect to the database using a generic application account. (Hopefully, you’ll have isolated it to an account used only for this purpose - if you haven’t, you should.)

OBI 11g : UPGAST-00055: error reading the Oracle Universal Installer inventory

Wed, 05 Oct 2011 00:00:00 +0000

It’s not my fault really.

When running an installation, presented with the option of

(a) do a bunch of stuff and wait to continue the install later or
(b) tick a box and continue now

it’s a better man that I who would opt for option (a).

When I recently installed OBIEE 11g, I was prompted to get a script run as root to set up the inventory, or tick “Continue Installation with local inventory” to continue with the install.

Oracle - tnsping - Message 3513 not found; product=NETWORK; facility=TNS

Mon, 26 Sep 2011 00:00:00 +0000

Short note to record this, as Google drew no hits on it.

Windows XP machine with existing Oracle 11.1 client installation, all working fine.

Installed Oracle 11.2 XE, and started getting these errors:

C:\Windows\System32>tnsping DBNAME

TNS Ping Utility for 32-bit Windows: Version 11.2.0.2.0 - Production on 26-SEP-2011 11:01:11

Copyright (c) 1997, 2010, Oracle.  All rights reserved.

Used parameter files:
C:\app\userid\product\11.1.0\client_1\network\admin\sqlnet.ora


Used TNSNAMES adapter to resolve the alias
Message 3513 not found;  product=NETWORK; facility=TNS
OK (20 msec)

Also got these errors from a previously-functioning ODBC query in Excel when I tried to refresh it:

Sourcecode markup tweaks in Wordpress

Mon, 26 Sep 2011 00:00:00 +0000

I noticed in Ed Stevens’ blog posting here that some sourcecode he’d posted had certain lines highlighted.

Wordpress provides the sourcecode tag for marking up sourcecode in blog posts. For example:

cd /some/random/folder ls -l # do not run this next line!

is much better presented as:

cd /some/random/folder ls -l # do not run this next line!

by wrapping it in

tags

I've known about the language='xx' attribute that you can use with the tag, but Ed's posting prompted me to check on the syntax and it turns out there a few tweaks one can use. Some of them are illustrated below. The list is taken from [Wordpress' Posting Source Code](http://en.support.wordpress.com/code/posting-source-code/) reference page.

## [sourcecode] ....

#REF: http://en.support.wordpress.com/code/posting-source-code/ # # This is some dummy source code to illustrate sourcecode posting on wordpress # Line 2 # Line 3 cd /some/random/folder ls -l # do not run this next line! rm -rf /some

....

#REF: http://en.support.wordpress.com/code/posting-source-code/ # # This is some dummy source code to illustrate sourcecode posting on wordpress # Line 2 # Line 3 cd /some/random/folder ls -l # do not run this next line! rm -rf /some

....

#REF: http://en.support.wordpress.com/code/posting-source-code/ # # This is some dummy source code to illustrate sourcecode posting on wordpress # Line 2 # Line 3 cd /some/random/folder ls -l # do not run this next line! rm -rf /some

....

#REF: http://en.support.wordpress.com/code/posting-source-code/ # # This is some dummy source code to illustrate sourcecode posting on wordpress # Line 2 # Line 3 cd /some/random/folder ls -l # do not run this next line! rm -rf /some

....

#REF: http://en.support.wordpress.com/code/posting-source-code/ # # This is some dummy source code to illustrate sourcecode posting on wordpress # Line 2 # Line 3 cd /some/random/folder ls -l # do not run this next line! rm -rf /some

....

#REF: http://en.support.wordpress.com/code/posting-source-code/ # # This is some dummy source code to illustrate sourcecode posting on wordpress # Line 2 # Line 3 cd /some/random/folder ls -l # do not run this next line! rm -rf /some

....

#REF: http://en.support.wordpress.com/code/posting-source-code/ # # This is some dummy source code to illustrate sourcecode posting on wordpress # Line 2 # Line 3 cd /some/random/folder ls -l # do not run this next line! rm -rf /some

....

#REF: http://en.support.wordpress.com/code/posting-source-code/ # # This is some dummy source code to illustrate sourcecode posting on wordpress # Line 2 # Line 3 cd /some/random/folder ls -l # do not run this next line! rm -rf /some

....

#REF: http://en.support.wordpress.com/code/posting-source-code/ # # This is some dummy source code to illustrate sourcecode posting on wordpress # Line 2 # Line 3 cd /some/random/folder ls -l # do not run this next line! rm -rf /some

Friday miscellany

Fri, 16 Sep 2011 00:00:00 +0000

If you only read one blog post this month, read James Morle’s eloquent attack on the term “Best Practice”.
I’m very excited to be joining RittmanMead next month! I’m looking forward to working with some of the industry’s most respected experts.

DBMS_STATS - GATHER AUTO

Tue, 13 Sep 2011 00:00:00 +0000

In Oracle 11g, the DBMS_STATS procedure GATHER_SCHEMA_STATS takes a parameter ‘options’ which defines the scope of the objects processed by the procedure call, as well as the action. It can be either GATHER or LIST (gather the stats, or list out the objects to be touched, respectively), and AUTO, STALE or EMPTY (defining the object selection to process).

GATHER on its own will gather stats on all objects in the schema
GATHER EMPTY / LIST EMPTY is self-explanatory - objects with no statistics.
GATHER STALE / LIST STALE is pretty obvious too - objects that have stale statistics (i.e. have had 10% change to them since statistics were last gathered). NB this 10% can be changed at an object/schema/DB level.
However, the documentation is ambiguous as to the precise function of GATHER AUTO / LIST AUTO.

There’s even a MOS note, “Differences between GATHER STALE and GATHER AUTO (Doc ID 228186.1)”, which strangely enough - given the precision of its title - doesn’t really explain the difference.

Using preupgrade to upgrade Fedora 14 to Fedora 15 - proxy errors

Mon, 12 Sep 2011 00:00:00 +0000

When using preupgrade to upgrade an existing Fedora 14 installation to Fedora 15, the following two errors were encountered:

Failed to fetch release info
No groups available in any repository

The box sits on a network behind a proxy out to the web.

The resolution was to make sure that environment variables http_proxy and https_proxy are set:

export http_proxy=http://user:password@proxyserver:port
export https_proxy=http://user:password@proxyserver:port

Make sure you do this from the user from which you run preupgrade.

Labelling Time axes in Excel

Thu, 08 Sep 2011 00:00:00 +0000

Excel may send chills down the spine of us when we hear users talking about its [ab]use, but it has its place in the toolset. For my money, it is a very good tool for knocking out graphs which look decent. Of course, rrdtool is my geek tool of choice for dynamic long-term graphing, but when doing scratch PoC work, I normally fall back to Excel.

One thing which has frustrated me over time is, well, time, and Excel’s handling thereof. How many times (these puns are getting tiresome already) have you seen an axis like this and gnashed your teeth?

A quotation to print out and stick on your wall

Thu, 18 Aug 2011 00:00:00 +0000

Here’s a quotation that I’ve just read and wanted to share. It is all part of a BAAG approach to troubleshooting problems, performance in particular.

From Greg Rahn (web|twitter) on oracle-l:

There are always exceptions, but exceptions can be justified and supported with data. Just beware of the the silver bullet syndrome…

The unfortunate part […] is that rarely anyone goes back and does the root cause analysis. It tends to fall into the bucket of “problem…solved”.

New blog from Oracle - OBI Product Assurance

Mon, 15 Aug 2011 00:00:00 +0000

Blogging from Oracle itself about OBIEE has always been a bit sparse, certainly in comparison to that which there is for core RDBMS.

It’s good to see a new blog emerge in the last couple of months from OBI Product Assurance, including some nice ’n spicy detailed config/tuning info.

Find it here: http://blogs.oracle.com/pa/.

There’s a couple more OBI blogs from Oracle, but both are fairly stale:

Implementing Oracle BI & EPM Solutions - Rob Reynolds
OBIEE Ramblings
Oracle BI Foundation (thanks Daan for pointing this one out)

Did you hear that thunk? That was me falling off my chair in shock

Mon, 08 Aug 2011 00:00:00 +0000

OK, a bit tired on a Monday morning, and so a bit sarcastic.

I’ve not really fallen off my chair, but I am shocked. I honestly didn’t think it would happen.

Oracle have finally released OBI 11g for HP-UX Itanium:

In other news, patchset 10.1.3.4.2 for OBI 10g was released today, I wonder if/when we’ll get an HP-UX Itanium version? The download page has it conspicuous by its absence even from “Coming Soon”:

Have you defined CLIENT_ID in OBIEE yet?

Mon, 08 Aug 2011 00:00:00 +0000

Have you defined CLIENT_ID in your OBIEE RPD yet? You really ought to.

As well as helping track down users of troublesome queries, it also tags dump files with the OBIEE user of an offending query should the worst occur:

For details, see:

OBIEE 10.1.3.4.2 released

Mon, 08 Aug 2011 00:00:00 +0000

A new version of OBI 10g (remember that?) has just been released, the Oracle twitter machine announced:

Along with presumably a bunch of bugfixes, the release notes list new functionality in catalog manager:

Download 10.1.3.4.2 from here

ODI 10g connectivity problem with OCI

Thu, 04 Aug 2011 00:00:00 +0000

Trying to connect to a repository in ODI using OCI. Target database is Oracle 11.1.0.7.

Throws this error:

com.sunopsis.sql.l: Oracle Data Integrator Timeout: connection with URL jdbc:oracle:oci8:@ODIPRD and user ODI_USER.
	at com.sunopsis.sql.SnpsConnection.a(SnpsConnection.java)
	at com.sunopsis.sql.SnpsConnection.t(SnpsConnection.java)
	at com.sunopsis.sql.SnpsConnection.connect(SnpsConnection.java)
	at com.sunopsis.tools.connection.DwgRepositoryConnectionsCreator.a(DwgRepositoryConnectionsCreator.java)
	at com.sunopsis.tools.connection.DwgRepositoryConnectionsCreator.a(DwgRepositoryConnectionsCreator.java)
	at com.sunopsis.graphical.l.oi.a(oi.java)
[...]

Normally this error would be caused by a misconfigured Oracle client. For example, a missing or incorrect tnsnames.ora entry. I validated these and got a successful response using tnsping.

Security issue on OBIEE 10.1.3.4.1, 11.1.1.3

Thu, 04 Aug 2011 00:00:00 +0000

July’s Critical Patch Update from Oracle includes CVE-2011-2241, which affects OBIEE versions 10.1.3.4.1 and 11.1.1.3. No details of the exploit other than it “allows remote attackers to affect availability via unknown vectors related to Analytics Server.”

It is categorised with a CVSS score of 5 (on a scale of 10), with no impact on Authentication, Confidentiality, or Integrity, and “Partial+” impact on Availability. So to a security-unqualified layman (me), it sounds like someone could remotely crash your NQSServer process, but not do any more damage than that.

Undocumented nqcmd parameters

Wed, 13 Jul 2011 00:00:00 +0000

I noticed on Nico’s wiki (which is amazing by the way, it has so much information in it) a bunch of additional parameters for nqcmd other than those which are displayed in the default helptext (nqcmd -h).

These are the additional ones:

-b
-w
-c
-n
-r
-t
-T (a flag to turn on time statistics)
-SmartDiff (a flag to enable SmartDiff tags in output)
-P
-impersonate 
-runas

Most parameters don’t appear to work in default call of nqcmd in 10g and 11g, throwing a Argument error near: error.

Oracle 11g - How to force a sql_id to use a plan_hash_value using SQL Baselines

Tue, 28 Jun 2011 00:00:00 +0000

Here’s a scenario that’ll be depressingly familiar to most reading this: after ages of running fine, and no changes to the code, a query suddenly starts running for magnitudes longer than it used to.

In this instance it was an ETL step which used to take c.1 hour, and was now at 5 hours and counting. Since it still hadn’t finished, and the gods had conspired to bring down Grid too (unrelated), I generated a SQL Monitor report to see what was happening:

Global statistics high/low values when using DBMS_STATS.COPY_TABLE_STATS

Wed, 15 Jun 2011 00:00:00 +0000

There is a well-documented problem relating to DBMS_STATS.COPY_TABLE_STATS between partitions where high/low values of the partitioning key column were just copied verbatim from the source partition. This particular problem has now been patched (see 8318020.8). For background, see Doug Burns’ blog and his excellent paper which covers the whole topic of statistics on partitioned tables.

This post Maintaining statistics on large partitioned tables on the Oracle Optimizer blog details what the dbms_stats.copy_table_stats does with regards to the high/low values:

Data Warehousing and Statistics in Oracle 11g - Automatic Optimizer Statistics Collection

Thu, 26 May 2011 00:00:00 +0000

Chucking a stick in the spokes of your carefully-tested ETL/BI …

My opinion is that automated stats gathering for non-system objects should be disabled on Oracle Data Warehouses across all environments.

All it does it cover up poor design or implementation which has omitted to consider statistics management. Once you get into the realms of millions or billions of rows of data, the automated housekeeping may well not have time to stat all of your tables on each run. And then it becomes a quasi-lottery when your tables will get processed. Or what if you’re working with intra-day loads (eg. near real-time) - the housekeeping job only runs once a day by default.

OBIEE performance - get your database sweating

Thu, 19 May 2011 00:00:00 +0000

Just because something produces the correct numbers on the report, it doesn’t mean you can stop there.

How you are producing those numbers matters, and matters a lot if you have an interest in the long-term health of your system and its ability to scale.

OBIEE is the case in point here, but the principle applies to any architecture with >1 tiers or components.

Let me start with a rhetorical question. The user has got a report which has ten rows of data. Which of the following methods is going to be a more efficient way to generate the report?

Entertaining Exadata FUD from HP

Mon, 11 Apr 2011 00:00:00 +0000

Chris Mellor at The Register posted an interesting article a couple of days ago, entitled HP and Violin build Oracle Exadata killer. The slidedeck has been removed from HP’s FTP site, but a bit of Google magic throws up a couple of mirror copies.

It’s an entertaining read (“Do a Proof of Concept! 94% win rate!! We can and do win against Exadata!!”), and a nice illustration of the FUD techniques that companies use in marketing their products against others. Greg Rahn has taken Netezza to task in the past for this, and to be fair at least Netezza had a serious white paper to back up their arguments. HP’s deck (including choice sections such as “How to sell against Exadata”) is IMHO nothing more than a biased set of arguments for salespeople to use to attempt to bullshit customers.

Oracle documentation - available on Kindle and iPad

Thu, 07 Apr 2011 00:00:00 +0000

Whilst perusing the Oracle database documentation, I noticed something which caught my eye:

As well as reading the documentation online as HTML or downloading as PDF for viewing on your computer etc, you can also download it in formats (Mobi and ePub) designed for eReaders such as the Kindle and iPad (the latter obviously isn’t “just” an eReader). For information on format support, there’s a handy table on Wikipedia.

It looks like the availability of mobi/epub files isn’t universal. For example, the OBI 11g documentation appears still to be HTML & ZIP only. And whilst the Database documentation doesn’t link to the mobi files on the contents page, only each document’s TOC, the Essbase documentation does:

Oracle XE 11gR2 installation - "OracleXEService should not be installed already"

Mon, 04 Apr 2011 00:00:00 +0000

Oracle XE 11gR2 beta has just been released, some details here and download here. It’s not a great deal of use for sandboxing DWH-specific stuff, given this list of excluded functionality (and this is by no means everything that’s not included):

Bitmapped index, bitmapped join index, and bitmap plan conversions

Oracle Partitioning

Parallel Data Pump Export/Import

Parallel query/DML

Parallel Statement Queuing

(source)

However, it’s always interesting to have to hand for trying out other things. And I like playing with new toys :)

Getting good quality I/O throughput data

Fri, 11 Mar 2011 00:00:00 +0000

This post expands on one I made last year here about sampling frequency (of I/O throughput, but it’s a generic concept). The background to this is my analysis of the performance and capacity of our data warehouse on Oracle 11g.

Before I get too boring, here’s the fun bit:

Pork Pies per Hour (PP/h)

Jim wants to enter a championship pork-pie eating competition. He’s timed himself practising and over the course of an hour he eats four pork pies. So we might say that his Pork Pies per Hour (PP/h) rate is 4.

Comparing methods for recording I/O - V$SYSSTAT vs HP Measureware

Wed, 09 Mar 2011 00:00:00 +0000

I wrote last year about Graphing I/O data using gnuplot and Oracle V$SYSSTAT, using a script from Kevin Closson in his article How To Produce Raw, Spreadsheet-Ready Physical I/O Data With PL/SQL. Good For Exadata, Good For Traditional Storage. Here I’ve got a simple comparison of the data recorded through this script (in essence, Oracle’s V$SYSSTAT), and directly on the OS through HP’s MeasureWare. It’s graphed out with my new favourite tool, rrdtool:

OBIEE Systems Management - dodgy counter behaviour

Tue, 08 Mar 2011 00:00:00 +0000

Over the last few months I’ve been doing a lot of exploring of OBIEE Systems Management data, covered in a mini-series of blog posts, Collecting OBIEE systems management data.

There are a vast number of counters exposed, ranging from the very interesting (Active Sessions, Cache Hits, etc) to the less so (Total Query Piggybacks, although for some seriously hardcore performance tuning even this may be of interest).

This short blog post is about a couple of counters which I’ve been monitoring but which looks to not be entirely reliable. Both are in the Oracle BI DB Connection Pool, and are:

Shiny new geek toys -- rrdtool and screen

Tue, 01 Mar 2011 00:00:00 +0000

I’ve added two new toys to my geek arsenal today. First is one with which I’ve dabbled before, but struggled to master. The second is a revelation to me and which I discovered courtesy of twitter.

rrdtool

rrdtool is a data collection and graphing tool which I’ve been aware of for a while. I wanted to use it when I wrote about Collecting OBIEE systems management data with JMX, but couldn’t get it to work. I’ll not lie to you - it is a bitch to work with at first. Or put a more polite way, it has a steep learning curve. But when you reach the top of the curve and realise its potential…wow. You’ll soon understand why it is so widely used. I plan to write this up soon, but it let me draw nice graphs like this:

Changing LDAP settings in an OBIEE RPD with UDML

Wed, 23 Feb 2011 00:00:00 +0000

A chap called Kevin posted a comment on a previous posting of mine asking

did you ever come across anything that could be used to change the LDAP server settings from a command line (admintool.exe, UDML, or otherwise)?

I did a quick play around with some UDML and it appears that you can.

Set up the initial LDAP server definition in the RPD

First I added a dummy LDAP server to samplesales.rpd:

Instrumenting OBIEE for tracing Oracle DB calls

Wed, 02 Feb 2011 00:00:00 +0000

Cary Millsap recently published a paper “Mastering Performance with Extended SQL Trace” describing how to use Oracle trace to assist with troubleshooting the performance of database queries. As with all of Cary Millsap’s papers it is superbly written, presenting very detailed information in a clear and understandable way. (and yes I do have a DBA crush ;-)) It discusses how you can automate the tracing of specific sessions on the database, and requiring the application to be appropriately instrumented. This reminded me of a post that I made almost exactly 12 months ago here, where I explained how to pass through the username of the OBIEE user to the database. Initially I thought it would be useful simply for being able to pin a rogue query to an end-user, but reading Cary’s paper made me realise there is more potential to it.

Materialised Views - PCT Partition Truncation

Sat, 08 Jan 2011 00:00:00 +0000

I’ve been doing some work recently that involved the use of Materialised Views on Oracle 11g (11.1.0.7), particularly around PCT refresh. There are some things that are not clear from the documentation, or are actually bugs so far as I’m concerned, and I’ve detailed these below.

In this example I was working on part of a DWH with c.2 millions rows aggregated up daily. One of the things that I spent a long time trying to get to work was Partition Truncation when using PCT refresh. We had tried and discarded “FAST” refresh as being completely non-performant for our volumes.

Oracle Whitepaper - "Best Practices for a Data Warehouse on Oracle Database 11g"

Wed, 05 Jan 2011 00:00:00 +0000

Poking around on My Oracle Support today, I found a link to a white paper dated November 2010, titled “Best Practices for a Data Warehouse on Oracle Database 11g”. It’s new to me and I’ve not noticed a blog post announcing it, so I thought I’d share it here. It’s by Maria Colgan, who has posted in the past on both the Inside the Oracle Optimizer blog and The Data Warehouse Insider blog. Here’s the link to it:

Data Warehousing and Statistics in Oracle 11g - incremental global statistics

Thu, 30 Dec 2010 00:00:00 +0000

This is a series of posts where I hope to humbly plug some gaps in the information available (or which has escaped my google-fu) regarding statistics management in Oracle 11g specific to Data Warehousing.

Incremental Global Statistics is new functionality in Oracle 11g (and 10.2.0.4?) and is explained in depth in several places including:

Oracle® Database Performance Tuning Guide - Statistics on Partitioned Objects
Greg Rahn - Oracle 11g: Incremental Global Statistics On Partitioned Tables
Inside the Oracle Optimiser - Maintaining statistics on large partitioned tables
Amit Poddar - One Pass Distinct Sampling (ppt - slides 52 onwards are most relevant)

In essence, Oracle maintains information about each partition when statistics is gathered on the partition, and it uses this to work out the global statistics - without having to scan the whole table. For a more detailed description, see the above links. It’s important to note that this is not the same as aggregated global statistics (which Doug Burns covers in detail here)

Adding OBIEE monitoring graphs into OAS

Mon, 06 Dec 2010 00:00:00 +0000

Introduction

This is the third part of three detailed articles making up a mini-series about OBIEE monitoring. It demonstrates how to capture OBIEE performance information, and optionally graph it out and serve it through an auto-updating webpage.

This final article describes how to bolt on to OAS a simple web page hosting the graphs that you created in part 2, plotting data from OBIEE collected in part 1.

The webpage

This is just an old-school basic HTML page, with a meta-refresh tag (which note that Chrome doesn’t work with) and img tags:

Charting OBIEE performance data with gnuplot

Mon, 06 Dec 2010 00:00:00 +0000

Introduction

This is the second part of three detailed articles making up a mini-series about OBIEE monitoring. It demonstrates how to capture OBIEE performance information, and optionally graph it out and serve it through an auto-updating webpage.

This article takes data that part one showed you how to collect into a tab-separated file that looks something like this:

2010-11-29-14:48:18     1       0       11      0       3       2       1       676     340     0       53      1       0       41      0       3       0
2010-11-29-14:49:18     1       0       11      0       3       2       1       676     0       0       0       1       0       0       0       3       0
2010-11-29-14:50:18     2       0       16      1       4       3       1       679     0       0       0       1       0       0       0       4       0
2010-11-29-14:51:18     2       2       19      1       4       3       1       679     32      0       53      1       0       58      0       4       0
2010-11-29-14:52:18     2       1       19      1       4       3       4       682     0       0       0       1       0       0       0       4       0
2010-11-29-14:53:18     2       1       19      1       4       3       4       682     0       0       0       1       0       0       0       4       0
2010-11-29-14:54:18     2       0       19      1       4       3       1       682     0       0       0       1       0       0       0       4       0

and plot it into something like looks like this:

Collecting OBIEE systems management data with jmx

Mon, 06 Dec 2010 00:00:00 +0000

Introduction

This is the first part of three detailed articles making up a mini-series about OBIEE monitoring. It demonstrates how to capture OBIEE performance information, and optionally graph it out and serve it through an auto-updating webpage.

For some background on OBIEE’s Systems Management component, along with JMX and MBeans, see here and here. The following assumes you know your mbeans from coffee beans and jmx from a bmx.

The metric collection is built around the jmxsh tool. This is similar to jmxterm and both provide command-line access to jmx. Once it’s commandline, it’s scriptable :)

OBIEE monitoring

Mon, 06 Dec 2010 00:00:00 +0000

Those of you who read my blog regularly may have noticed I have a slight obsession with the OBIEE systems management capability which is exposed through JMX. Venkat has blogged this week about JMX in OBI11g, and it’s clearly a technology worth understanding properly. I’ve recently been tinkering with how to make use of it for monitoring purposes, most recently using JConsole and discussed here. What follows is an extension of this idea, cobbled together with a bit of shell scripting, awk, gnuplot, and sticky backed plastic. It’s built on OBIEE 10g - for OBI11g it may differ (although I understand that Performance MBeans still exist).

OBIEE 10g - javahost hang

Fri, 03 Dec 2010 00:00:00 +0000

Hot on the heels of one problem, another has just reared its head.

Users started reporting an error with reports that included charts:

Chart server does not appear to be responding in a timely fashion. It may be under heavy load or unavailable.

Set up is a OBIEE 10.1.3.4.1 two-server deployment with BI/PS/Javahost clustered and loadbalanced throughout.

Diagnostics

Javahost was running, and listening, on both servers:

$ps -ef|grep javahost
obieeadm 14076     1  0  Nov 25  ?         9:23 /app/oracle/product/OracleAS_1/jdk/bin/IA64N/java -server -classpath /app/oracle/product/obiee/web/javahost/lib/core/sautils.ja
$netstat -a|grep 9810|grep LISTEN
tcp        0      0  *.9810                 *.*                     LISTEN

In Javahost log file on both servers there were these errors reported, but since javahost had started over a week ago:

Troubleshooting OBIEE - LDAP (ADSI) authentication

Thu, 02 Dec 2010 00:00:00 +0000

They say about travelling that it’s the journey and not the destination, and the same is true with this problem we hit during a deployment to Production.

We were deploying a new OBIEE 10g implementation, with authentication provided by Microsoft Active Directory (AD) through the LDAP functionality in OBIEE. As a side note, it’s a rather nice way to do authentication, although maybe I’m biased coming from our previous implementation which used EBS integrated authentication and was a bugger to set up and work with.

A Poor Man's OBIEE EM/BI Management Pack

Thu, 04 Nov 2010 00:00:00 +0000

Folk from Yorkshire are tight, so the stereotype goes. So here’s a cheap-ass way to monitor OBIEE 10g using nothing but the OBIEE built-in systemsmanagement component, the jmx agent, and jconsole (which is part of the standard Java distribution):

From here you can also export to CSV the various counters, and then store history, plot it out with gnuplot or Excel, etc.

If anyone’s interested let me know and I’ll document a bit more about how I did this, but it’s basically building on previous work I’ve documented around jmx and OBIEE.

Analysing ODI batch performance

Wed, 03 Nov 2010 00:00:00 +0000

I’ve been involved with some performance work around an ODI DWH load batch. The batch comprises well over 1000 tasks in ODI, and whilst the Operator console is not a bad interface, it’s not very easy to spot the areas consuming the most runtime.

Here’s a set of SQL statements to run against the ODI work repository tables to help you methodically find the steps of most interest for tuning efforts.

Does this summarise your system development & support ethos?

Wed, 27 Oct 2010 00:00:00 +0000

I heard this on Thinking Allowed and thought how applicable it was to the attitudes that you can sometimes encounter in both systems development, and the support of production systems:

“Each uneventful day that passes reinforces a steadily growing false sense of confidence that everything is all right – that I, we, my group must be OK because the way we did things today resulted in no adverse consequences.”

by Scott Snook (Senior Lecturer in the Organizational Behavior unit at Harvard Business School )

Two excellent OBI presentations from Jeff McQuigg

Wed, 27 Oct 2010 00:00:00 +0000

Jeff McQuigg has posted two presentations that he gave at Openworld 2010 on his website here: http://greatobi.wordpress.com/2010/10/26/oow-presos/

They’re full of real content and well worth a read. There’s excellent levels of detail and plenty to think about if you’re involved in OBI or DW development projects.

Graphing I/O data using gnuplot and Oracle V$SYSSTAT

Tue, 26 Oct 2010 00:00:00 +0000

Continuing in the beard-scratching theme of Unix related posts (previously - awk), here’s a way to graph out the I/O profile of your Oracle database via the Oracle metrics in gv$sysstat, and gnuplot. This is only the system I/O as observed by Oracle, so for belts & braces (or to placate a cynical sysadmin ;-)) you may want to cross-reference it with something like sar.

First, a pretty picture of what you can get:

awk - split a fixed width file into separate files named on content

Tue, 19 Oct 2010 00:00:00 +0000

More of a unix thing than DW/BI this post, but I have a beard so am semi-qualified….

The requirement was to improve the performance of some ODI processing that as part of its work was taking one huge input file, and splitting it into chunks based on content in the file. To add some (minor) spice the file was fixed width with no deliminators, so the easy awk answers that I found on google weren’t applicable.

When is a bug not a bug? When it's a "design decision"

Mon, 18 Oct 2010 00:00:00 +0000

Last month I wrote about a problem that Informatica as part of OBIA was causing us, wherein an expired database account would bring Oracle down by virtue of multiple connections from Informatica.

I raised an SR with Oracle (under OBIA support), who after some back-and-forth with Informatica, were told:

This is not a bug. That the two error messages coming back from Oracle are handled differently is the result of a design decision and as such not a product fault.

A good maxim to bear in mind when designing reports

Thu, 23 Sep 2010 00:00:00 +0000

Jeff Atwood of Coding Horror fame observed:

This is sage advice (if a little crude :-)) to bear in mind when building reports for users.

Better safe than sorry...sanitising DB input

Wed, 22 Sep 2010 00:00:00 +0000

As Twitter learnt yesterday, you should always sanitise user input. I was amused to see My Oracle Support doing so….recursively :)

The apostrophe in “doesn’t” got escaped once, and then again, and then again, and then again, and then again ……

TortoiseSVN doesn't prompt for authentication

Tue, 21 Sep 2010 00:00:00 +0000

Here’s one in the series of stupid things I’ve done but which Google has thrown no answers, so I post it here to help out fellow idiots.

Today’s episode involves our SCM tool, TortoiseSVN. I’d been happily using it for over a year, when suddenly I couldn’t commit any more. I could browse and checkout to my heart’s content, but when I tried to commit, boom:

Commit failed (details follow): Authorization failed

The danger of averages - Measuring I/O throughput

Tue, 14 Sep 2010 00:00:00 +0000

This query, based on AWR snapshots on sys.wrh$_sysstat, includes in its metrics the I/O read throughput for a given snapshot duration.

However it’s important to realise the huge limitation to this figure - it’s an average. It completely shoots you in the foot if you’re looking at capacity requirements.

Consider this real-life example extracted from the above query:

Timestamp					Total Read MBPS
===========================================
14-SEP-10 05.15.12.660      113.748
14-SEP-10 06.00.40.953      202.250
14-SEP-10 06.45.52.750       34.649
14-SEP-10 07.30.03.394       10.953
14-SEP-10 08.15.15.243       57.833
14-SEP-10 09.00.27.180       30.177

So, it looks like early in the morning we’re using about 200 MB/s throughput, and by about 9am somewhere around 30-50 MB/s ?

RTFM? But where TF is the FM? >> Offline searchable OBIEE 11g documentation

Mon, 13 Sep 2010 00:00:00 +0000

I’m a geek. I like understanding things in their absolute entirety. It frustrates me to have to make presumptions or assumptions about something. I like to get down ’n dirty and find out what makes things tick.

So that necessitates reading manuals. And unfortunately, Oracle don’t make that easy all the time.

Some of the manuals are not very good, and others are difficult to find. Given the complexity of the OBIEE stack and proliferation of terminology and product names, sorting the wheat from the chaff can be a headache. Fair enough, for a corporation as big as Oracle with a product portfolio in the [hundreds? thousands?] creating a unified, comprehensive, easy-to-navigate point of reference for documentation must be near on impossible - but that’s their problem to figure out and mine to bitch about until it’s done ;-)

A fair bite of the CPU pie? Monitoring & Testing Oracle Resource Manager

Fri, 10 Sep 2010 00:00:00 +0000

Introduction

We’re in the process of implemention Resource Manager (RM) on our Oracle 11gR1 Data Warehouse. We’ve currently got one DW application live, but have several more imminent. We identified RM as a suitable way of - as the name would suggest - managing the resources on the server.

In the first instance we’re looking at simply protecting CPU for, and from, future applications. At some point it would be interesting to use some of the more granular and precise functions to demote long-running queries, have nighttime/daytime plans, etc. I’d also like to explore the management of IO but for us the pain is in bandwidth that a query consumes, and it looks like RM can only work with total session MB, or IOPS. Reading about Exadata it sounds like the Exadata I/O Resource Management might do this ([…]It allows intra and inter-database I/O bandwidth to be defined and managed[..]). But for that I’ll have to write to Santa and promise to be a good boy this year.

Misbehaving Informatica kills Oracle

Thu, 02 Sep 2010 00:00:00 +0000

This problem, which in essence is bad behaviour from Informatica bringing down Oracle, is a good illustration of unintended consequences of an apparently innocuous security setting. Per our company’s security standards, database passwords expire every 90 days. When this happens users are prompted to change their password before they can continue logging into Oracle. This applies to application user IDs too. It appears that Informatica 8.6.1 HF6 (part of OBIA 7.9.6.1) doesn’t handle an expired password well, spawning multiple connections to the database, eventually bringing Oracle down through memory SWAP space exhaustion.

BI Publisher - error creating Quartz tables

Wed, 25 Aug 2010 00:00:00 +0000

A very short blog post to break the drought, but I didn’t hit any google results for this error so thought it might be useful to record it.

In BI Publisher 10.1.3.4, trying to install the Scheduler (Quartz) schema, I got this error:

Schema installation failed while creating tables. Schema may already exist. Please remove the existing schema or choose another database and try again.

To me, the error text is a bit unhelpful. Whilst the first statement is correct - “Schema installation failed while creating tables”, it doesn’t tell you the error it encountered, and then it goes on to suggest only one reason for the failure.

Measuring real user response times for OBIEE

Mon, 14 Jun 2010 00:00:00 +0000

@alexgorbachev tweeted me recently after picking up my presentation on Performance Testing and OBIEE.

His question got me thinking, and as ever the answer “It Depends” is appropriate here :-)

Why is the measurement being done?

Without knowing the context of the work Alex is doing, how to measure depends on whether the measurement needs to be of: -

The actual response times that the users are getting, or
The response times that the system is currently capable of delivering

This may sound like splitting hairs or beard-scratching irrelevance, but it’s not. If the aim of the exercise is to be able to make a statement along the lines of:

Scripts to extract information from OBIEE NQQuery.log

Fri, 11 Jun 2010 00:00:00 +0000

Here are a couple of little unix scripts that I wrote whilst developing my performance testing OBIEE method.

They’re nothing particularly special, but may save you the couple of minutes it’d take to write them :)

Note that some of this data is available from Usage Tracking and where it is I’d recommend getting it from there, databases generally being easier to reliably and repeatably query than a transient log file.

OBIEE 11g launch date - 7th July 2010

Thu, 03 Jun 2010 00:00:00 +0000

OBIEE 11g is going to be officially launched on 7th July (this year!) in London: Launch Event: Introducing Oracle Business Intelligence Enterprise Edition 11g

h/t to Twitter and blogs (including an Oracle-branded one) this morning. Good news travels fast! :-)

Looks like John Minkjan won the bet, did he know something we didn’t? ;-) (although is “Launch” the same as “GA”?)

Performance Testing and OBIEE

Mon, 24 May 2010 00:00:00 +0000

Here’s my presentation “Performance Testing and OBIEE” that I gave at the RittmanMead BI Forum 2010: Performance Testing and OBIEE.pptx

It’s a Powerpoint 2007 file (pptx) for which you may need the Microsoft PowerPoint Viewer 2007. I’ve included copious notes on each slide which hopefully cover the gist of what I talked about when I was delivering it. There are also a handful of funky animations which is why I’ve left it in pptx and not exported to PDF or other format (sorry Open Office users).

My first presentation - afterthoughts

Sun, 23 May 2010 00:00:00 +0000

I delivered my first presentation today, at the RittmanMead BI Forum.I was really nervous in the hours and minutes leading up to it, but once I got up there and started talking I actually quite enjoyed it. If you were in the audience, I’d love some feedback in the comments section below, particularly any “constructive criticism”. I obviously didn’t make too much of a mess of it, as I was awarded “best speaker” of the event, which was a great honour. Hopefully I’ll get the opportunity to present again soon, perhaps at an UKOUG event.

RittmanMead BI Forum 2010

Fri, 21 May 2010 00:00:00 +0000

I’ve just returned from the RittmanMead 2010 BI Forum which this year was at the Seattle Hotel in Brighton Marina.

The event was limited to 50 attendees, and I think this was a good number. The event was explicitly pitched at a very technical expert level, and the audience very much represented this. Whereas at a larger conference you may find the odd manager wandering around pretending to be technical ;-) this event was most definitely not pitched at such types.

Validating EBS-BI authentication, without BI

Mon, 17 May 2010 00:00:00 +0000

Troubleshooting EBS-BI integrated authentication can be a tiresome activity, so here’s a shortcut that might help. If you suspect the problem lies with EBS then you can leave OBIEE out of the equation.

Login to EBS
Use FireBug or Fiddler2 to inspect web traffic as follows:
Click the BI link from EBS
Should be first a request to EBS server, which returns 302 and redirects to http://<bi server>:<port>/analytics/saw.dll?Dashboard&acf=101507310
Record the value of acf (eg 101507310)

What am I missing here??? ORA-01017: invalid username/password; logon denied

Thu, 06 May 2010 00:00:00 +0000

What’s going on here? The username/password is definitely valid, proved by the sqlplus connection.

Configuring DAC in OBIA 7.9.5.1:

What can I do for you?

1 - Enter repository connection information
2 - Test repository connection
3 - Enter email account information
4 - Send test email
5 - Save changes
6 - Exit

Please make your selection: 1

These are your connection type choices:

1 - MSSQL
2 - DB2
3 - Oracle (OCI8)
4 - Oracle (Thin)
5 - Keep current ( Oracle (Thin) )

Please make your selection: 4

Current value for Instance is MYDB.
Press return to keep it or enter a new value.
> MYDB

Current value for Database Host is server.company.com.
Press return to keep it or enter a new value.
> server.company.com

Current value for Database Port is 1521.
Press return to keep it or enter a new value.
> 1521

Current value for Table owner name is DAC_REPO_795.
Press return to keep it or enter a new value.
> DAC_REPO_795

Press return to keep current password, enter a new value otherwise.
> HAS425Al

What can I do for you?

1 - Enter repository connection information
2 - Test repository connection
3 - Enter email account information
4 - Send test email
5 - Save changes
6 - Exit

Please make your selection: 2

Connecting to repository...
Can't connect to the database.
ORA-01017: invalid username/password; logon denied

Validate connectivity with SQLPLUS:

RTFAL!

Sat, 24 Apr 2010 00:00:00 +0000

This is a note-to-self really. When playing around with Oracle and something’s not working - RTFAL: Read The Flippin Alert Log!

After resizing a VM I was getting this problem:

[oracle@RNMVM01 ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.1.0 Production on Sat Apr 24 17:44:44 2010

Copyright (c) 1982, 2009, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup nomount
ORA-00845: MEMORY_TARGET not supported on this system
SQL>

I spent longer than I should have reading around on google, hitting various pages which all talked around memory management and /dev/shm

Opera + Oracle EM = true love

Thu, 22 Apr 2010 00:00:00 +0000

A faithful FireFox user for many users, I’ve been having a rather delightful rekindling of my old love for Opera recently. Back in the days, when I was a beardless bairn, I even paid for Opera I liked it so much. Then Firefox came along and Opera got dropped by the wayside like a teenage crush.

Well Opera 10.5 was released recently and I’m liking it. I miss my Firefox extensions too much to switch entirely, but damn, Opera is FAST!!

My first presentation - help!

Wed, 21 Apr 2010 00:00:00 +0000

I’m doing my first ever conference presentation next month at the 2010 Rittman Mead BI Forum.

My presentation is called Performance Testing OBIEE, which is something I’ve spent a lot of time working on over the last few months. I think the challenge is going to be distilling it all into a session that’s not going to overwhelm everyone or bore them to death! Well, actually, the challenge is going to be the presenting. I can talk geek one-on-one, but talking to a whole bunch of people, not wittering but staying focussed, holding their attention….uh oh.

OBIEE 11g tidbit - XUDML support

Thu, 18 Mar 2010 00:00:00 +0000

Spotted this when trawling through My Oracle Support. It’s pretty common knowledge anyway amongst people already familiar with hacking around with OBIEE, but worth recording for people coming along to it new.

Doc ID 1068266.1 states:

UDML is not supported in OBI 10g.

in 11g, XUDML (the Oracle BI Server XML API) will be fully supported and documented.

ORA-13757: "SQL Tuning Set" "string" owned by user "string" is active.

Tue, 09 Mar 2010 00:00:00 +0000

I’ve been playing around with SQL Tuning Sets, and was trying to clear up my mess.

To list all the tuning sets:

SET WRAP OFF
SET LINE 140
COL NAME FOR A15
COL DESCRIPTION FOR A50 WRAPPED

select name,created,last_modified,statement_count,description 
from DBA_SQLSET

NAME            CREATED   LAST_MODI STATEMENT_COUNT DESCRIPTION
--------------- --------- --------- --------------- ----------------------------------------------------------------------------------------
sts_test_02     09-MAR-10 09-MAR-10               1 Test run 1
sts_test_01     12-FEB-10 12-FEB-10               1 an old STS test test test

To delete a tuning set:

Securing OBIEE Systems Management JMX for remote access

Fri, 05 Mar 2010 00:00:00 +0000

JMX

OBIEE’s Systems Management functionality exposes performance counters and the application’s configuration options through Java MBeans and optionally a protocol called JMX.

It’s extremely useful, and is documented pretty widely :

In this article I’m going to discuss the use of JMX to access these counters remotely, and a possible security issue that’s present in the BI Management Pack manual. The BI Management Pack is an add-on to Oracle’s Enterprise Manager / Grid Control for managing OBIEE, see Mark Rittman’s excellent guide on Oracle’s website.

Who's been at the cookie jar? EBS-BI authentication and Load Balancers

Fri, 05 Mar 2010 00:00:00 +0000

We hit a very interesting problem in our Production environment recently. We’d made no changes for a long time to the configuration, but all of a sudden users were on the phone complaining. They could login to BI from EBS but after logging in the next link they clicked took them to the OBIEE “You are not logged in” screen.

Our users login to EBS R12 and then using EBS authentication log in to OBIEE (10.1.3.4). Our OBIEE is deployed on OAS, load balanced across two servers by an F5 BIG-IP hardware load balancer.

OBIA 7.9.6 Performance Recommendations

Tue, 02 Mar 2010 00:00:00 +0000

A new document has been published by Oracle, discussing ways of improving performance for OBIA 7.9.6 and 7.9.6.1. Its primary focus is around improving ETL performance. There’s some very interesting content including hardware sizing recommendations, and I’d strongly recommend anyone working with OBIA reads it.

It’s called “Oracle Business Intelligence Applications Version 7.9.6 Performance Recommendations” and is available on My Oracle Support through Doc ID 870314.1

Oracle Support Blog back online, kinda.

Tue, 23 Feb 2010 00:00:00 +0000

After it was ripped down last week, Chris Warticki’s blog is back online, albeit with the last posting redirecting visitors to a new location on Oracle Communities.

Maybe I’m getting too old for this s##t, but I’m yet to really get a handle on how Oracle want to interact with real people on the ground. Oracle Communities is a fairly new site that I’ve not explored so much because of no OBIEE area. My Oracle Support has feedback and comment sections, but in general usability stinks. A support blog is torn down and then reappears. Oracle Forums are popular as ever but the software blows and some forums (eg OBIEE) are withering under a very high noise to quality-content ratio. Oh, and Oracle Mix (“Bringing Oracle Customers, Employees, and Developers together”) too, which looked quite neat but I have no idea how that’s supposed to fit it the picture.

Oracle Support blog no more

Mon, 15 Feb 2010 00:00:00 +0000

A sad little passing last week, of the Oracle Support Blog and related tweets by Chris Warticki.

Last week Chris posted this comment on twitter

“So, what to do if you’re the “online customer presence” and your own leadership wants to censor your posts and comments?”

followed by this terse blog posting:

Support Blog: No longer available

By chris.warticki on February 12, 2010 4:08 PM Please use My Oracle Support Communities instead

Exadata V2 POC numbers

Wed, 10 Feb 2010 00:00:00 +0000

I stumbled across this blog posting recently, and am reposting the link here as I’ve seen no mention of it elsewhere. This surprised me as with Exadata (and most new technologies) any snippet of news or tech insight gets blogged and tweeted multiple times over.

Any insight into Exadata is interesting as unlike all the other Oracle software which can be downloaded and poked & prodded, this is a physical box so we rely on blog postings and marketing (with a BS filter) to understand more about it.

Illustrating data

Mon, 08 Feb 2010 00:00:00 +0000

In the following list, which two mind-mapping programs are rated best?

Now look at the actual numbers, and answer again

Different answer?

I can’t be the only one in this frantic world whose eyes are drawn to the pictures instead of words and leap to conclusions. It’s only because I use FreeMind and was surprised it scored so low …. and then realised it hadn’t. Looks like the HTML rendering isn’t the same here (FF3.6) as when the web page author wrote it.

Brilliant performance articles by Cary Millsap

Fri, 29 Jan 2010 00:00:00 +0000

There is a LOT written about performance. And this post is now adding to it. Some of it’s excellent, some of it less so. But a lot of it starts from a point so far down the process that unless you know the first bit, you’re going to raring off and end up chasing your tail or p###ing in the wind…. (pardon my french). Without a well structured approach that you understand and always follow you’ll hit on solutions by luck only.

Identify your OBIEE users by setting Client ID in Oracle connection

Tue, 26 Jan 2010 00:00:00 +0000

You get a call from your friendly DBA. He says the production database is up the spout, and it’s “that bee eye thingumy causing it”. What do you do now? All you’ve got to go on is a program name in the Oracle session tables of “nqsserver@MYSERVER (TNS V1-V3)” and the SQL the DBA sent you that if you’re lucky will look as presentable as this: The username against the SQL is the generic User ID that you had created for connections to the database from OBIEE.

How to resolve "[nQSError: 12002] Socket communication error at call=: (Number=-1) Unknown"

Fri, 22 Jan 2010 00:00:00 +0000

This error caught me out today. I was building a Linux VM to do some work on, and for the life of me couldn’t get the OBIEE Admin Tool to connect to the BI Server on the VM.

The error I got when trying to define a DSN on the Windows box was:

[nQSError: 12008] Unable to connect to port 9703 on machine 10.3.105.132 [nQSError: 12010] Communication error connecting to remote end point: address = 10.3.105.132; port = 9703. [nQSError: 12002] Socket communication error at call=: (Number=-1) Unknown

Hardening OAS

Thu, 21 Jan 2010 00:00:00 +0000

Oracle Application Server (OAS) is the Web and Application server typically deployed with OBIEE. There are several settings which by default may be viewed as security weaknesses. Whether realistically a target or not, it’s good practice to always be considering security and lock down your servers as much as reasonably possible. I adopt the default stance of having to find a reason to leave something less secure, rather than justify why it needs doing.

libnnz10.so: cannot restore segment prot after reloc: Permission denied

Fri, 18 Dec 2009 00:00:00 +0000

Quick post as the snow’s coming down and I wanna go home …

I’ve been working on building a VM based on OEL5.4 and OBIEE 10.1.3.4.1. After installing XE 10.2 I tried to fire my RPD up, but hit this:

/usr/lib/oracle/xe/app/oracle/product/10.2.0/server/lib/libnnz10.so: cannot restore segment prot after reloc: Permission denied [nQSError: 46029] Failed to load the DLL /app/oracle/product/obiee/server/Bin/libnqsdbgatewayoci10g.so. Check if ‘Oracle OCI 10G’ database client is installed.

If you trace the ‘stack’ back you find that it parses down to this nub of an error:

Running the OBIEE admin tool on Unix

Mon, 14 Dec 2009 00:00:00 +0000

Mucho kudos to Borkur Steingrimsson for getting the OBIEE admin tool working on Unix!

CAF

Fri, 11 Dec 2009 00:00:00 +0000

Very interesting post by Kevin McGinley about CAF here: CAF = Migration Utility? Use Caution!

It articulates better than I ever could reasons against using CAF particularly in a production environment.

Since the tool came out I’d been struggling to get my head around it, convinced I was missing something. I still don’t profess to understand it properly, but Kevin’s article reassures me that I shouldn’t be losing too much sleep over it, especially given that it’s unsupported and won’t work with 11g.

Troubleshooting Presentation Services / analytics connectivity

Wed, 09 Dec 2009 00:00:00 +0000

Short but sweet this one - a way of troubleshooting connectivity problems between analytics (the Presentation Services Plug-in, either j2ee servlet or ISAPI, a.k.a. SAWBridge) and sawserver (Presentation Services).

For a recap on the services & flow please see the first few paragraphs of this post.

Problems in connectivity between analytics and sawserver normally manifest themselves through this error message:

500 Internal Server Error Servlet error: An exception occurred. The current application deployment descriptors do not allow for including it in this response. Please consult the application log for details.

UKOUG TEBS 2009

Thu, 03 Dec 2009 00:00:00 +0000

This was my first UKOUG TEBS, in fact my first conference I’d ever attended! I was quite unsure what to expect, but three days later I can safely say it was invaluable.

The variety of presentations and expertise being shared was impressive, and it was great to hear people sharing and discussing their ideas and opinions around the subjects I work with each day.

Working in isolation is not a good idea, one can develop a blinkered or bunker mentality. Blogging is good for breaking out of this, as is, I’m now convinced, twittering. However, nothing beats meeting up with folk and discussing problems face to face, listening and learning from their experience.

I think this summarises everything.

Fri, 27 Nov 2009 00:00:00 +0000

Why did this make me think of the OBIA upgrade documentation?? ;-)

via I think this summarizes everything..

[Update December 18, 2013: The lady in the picture is Kathy Sierra]

OBIEE application servers, now and future

Wed, 25 Nov 2009 00:00:00 +0000

Oracle have published an interesting doc 968223.1, entitled “Enterprise Deployment of Oracle BI EE on OC4J and App Servers”.

It details the differences between OC4J and OAS which is useful for the current versions of OBIEE. It then also gives a useful heads-up – that WebLogic becomes the App server of choice in the next version of OBIEE

All of this changes in OBI EE 11g where several projects will become absolutely dependent upon an App Server. Within Oracle Fusion Middleware, WebLogic Server will be the application server and OBI EE 11g will be deployed and certified with WebLogic Server.

SoOotW and sweep

Wed, 25 Nov 2009 00:00:00 +0000

(Sorry for the lame title, but it gives me an excuse to put this picture in :) )

I was really pleased by the response I got from my posting about The state of OBIEE on the web, knowing that it’s not just me goes a long way to keeping my blood pressure down.

The OTN forum is still an unbalanced mix of tosh (is this guy for real?) with the occasional insightful and interesting post or idea such as this really neat one from Joe Bertram about using multiple presentation services on top of the same RPD to give different interfaces to different end devices.

Resolved: sawserver : Error loading security privilege /system/privs/catalog/ChangePermissionsPrivilege

Tue, 17 Nov 2009 00:00:00 +0000

Whilst installing OBIA 7.9.6.1 I hit this problem when firing up Presentation Services (sawserver):

Error loading security privilege /system/privs/catalog/ChangePermissionsPrivilege.

A quick search on the forums threw up two posts suggesting a corrupted WebCat.

Since I’d got this webcat fresh out of the box I was puzzled how it could be corrupted.

I did a bit more tinkering (including nosying around in the sawserver log), before realising it was indeed corrupt, and that I was indeed a muppet.

Deploying Oracle Business Intelligence Enterprise Edition on Sun Systems

Thu, 12 Nov 2009 00:00:00 +0000

A very interesting new PDF from Sun on deploying OBIEE has been published, with discussions on architecture, performance and best practice.

This Sun BluePrints article describes an enterprise deployment architecture for Oracle Business Intelligence Enterprise Edition using Sun servers running the Solaris Operating System and Sun Storage 7000 Unified Storage systems. Designed to empower employees in organizations in any industry—from customer service, shipping, and finance to manufacturing, human resources, and more—to become potential decision makers, the architecture brings fault tolerance, security, resiliency, and performance to enterprise deployments. Taking advantage of key virtualization technologies, the architecture can be used to consolidate multiple tiers onto a single system to help reduce cost and complexity. A short discussion of the performance characteristics of the architecture using a realistic workload also is included.

#Fail: My Oracle Support

Wed, 11 Nov 2009 00:00:00 +0000

Metalink was retired this weekend and made way for the new My Oracle Support system. It didn’t go as smoothly as it could have done.

This post is going to be a bit of a rambling rant.

Ultimately people, including me, don’t like their cheese being moved (not unless there’s a really runny piece of Camembert at the end of it). That makes it a bit more difficult to discuss because some of people’s complaints will just be geeks being stubborn (and boy, can geeks be stubborn). Arguments descend into minutiae of detail and flash vs DHTML - whilst the bigger picture gets lost.

OBIEE clustering - specifying multiple Presentation Services from Presentation Services Plug-in

Fri, 06 Nov 2009 00:00:00 +0000

Introduction

Whilst the BI Cluster Controller takes care nicely of clustering and failover for BI Server (nqsserver), we have to do more to ensure further resilience of the stack.

A diagram I come back to again and again when working out configuration or connectivity problems is the one on P16 of the Deployment Guide. With this you can work out most issues for yourself through simple reasoning. Print it out, pin it to your wall, and read it!

Advanced Googling for OBIEE information

Tue, 03 Nov 2009 00:00:00 +0000

Want to find all PDFs from Oracle about OBIEE?

There’s some interesting ones that this turned up on Oracle’s public FTP. In particular:

(There’s also PowerPoint files on there including this one but Google doesn’t seem to index them)

You can use Google’s Advanced Search page to build similar queries:

CAF installation video

Tue, 03 Nov 2009 00:00:00 +0000

Christian Screen has done a nice video explaining the CAF installation, and has promised a deep-dive followup which I’m looking forward to. Click here for the article

Experts

Mon, 02 Nov 2009 00:00:00 +0000

A brilliant posting here from Jonathan Lewis on the subject of Experts. He in turn is quoting Chen Shapira: “DBAs are under a lot of pressure not to be experts.”. Read the sentence again, as it took me a minute to figure out. He’s writing in the context of an Oracle DBA but I think it’s equally applicable to those working with an looking after installations of OBIEE et al.

The state of OBIEE on the web

Fri, 30 Oct 2009 00:00:00 +0000

An advance footnote

I’ll start this by saying why I think things are how they are, and then I’ll get to the meat of my article.

OBIEE in its current incarnation (v10.1.3) is a mature product. All the big bugs have been caught and fixed. All the known quirks are well documented. All the missing features are known. All the clever workarounds have been found. All the neat little hacks have been explored.

The mind boggles...

Mon, 26 Oct 2009 00:00:00 +0000

I honestly don’t dare click the “Did you mean” link ;-)

Troubleshooting OBIEE and ORA-12154: TNS:could not resolve the connect identifier

Thu, 22 Oct 2009 00:00:00 +0000

A frequent question on the OTN OBIEE forum is how to fix this error:

[nQSError: 17001] Oracle Error code: 12154, message: ORA-12154: TNS:could not resolve the connect identifier specified at OCI call OCIServerAttach. [nQSError: 17014] Could not connect to Oracle database.

The error is simply OBIEE reporting that it tried to connect from the BI Server to an Oracle database and the Oracle client returned an error. Distilling it down gives us this error:

Critical Patch Update - OBIEE vuln CVE-2009-1990

Wed, 21 Oct 2009 00:00:00 +0000

October’s Oracle Critical Patch Update Advisory has been released. There are two vulnerabilities (CVE-2009-1999, CVE-2009-1990) listed under Oracle Application Server for “Component” Business Intelligence Enterprise Edition and one (CVE-2009-3407) for “component” Portal.

CVE-2009-1999 is OBIEE and “Fixed in all supported versions. No patch provided in this Critical Patch Update.”.
CVE-2009-3407 looks like only OAS (not OBIEE), up to versions 10.1.2.3 and 10.1.4.2.
CVE-2009-1990 is OBIEE and is the main vuln of interest. It’s unclear if it’s just OBIEE 10.1.3.4.x, or all versions of OBIEE through to and including 10.1.3.4.1. It’s also confusing putting it on the same table as OAS especially given it has similar versioning (10.1.3.x.x).

For information about patches, see My Oracle Support Note 881382.1. This doc lists patches 8927890 and 8927886 for OBIEE 10.1.3.4.1 and 10.1.3.4.0 respectively. Since no other versions are mentioned that suggests it doesn’t affect them but that’d be a heck of an assumption to make and if I were running < 10.1.3.4.0 I’d be raising an SR to seek clarification especially given the ambiguity of the table in the Advisory doc.

OTN forum - "Pro"

Wed, 21 Oct 2009 00:00:00 +0000

w00t :-D

Yesterday I got my “Pro” medal for 500 points on OTN Forums

I’d been intending to post a grumpy rant about OTN recently, but maybe I’ll postpone that for a few days now ;)

New OBIEE blogs

Mon, 19 Oct 2009 00:00:00 +0000

Two new blogs of note :

Both are well worth a read & following.

BI Server hung - nQSError 14054 / 15001 / 23005

Fri, 16 Oct 2009 00:00:00 +0000

Watch out if you are using init blocks in your RPD. We hit a bug (#9019374) recently that caused BI Server (10.1.3.4) to hang.

The init block in question should have returned a date to update a repository variable, but because of badly-written SQL and abnormal data in the source table actually returned a null value. BI Server evidently didn’t like this null being inserted somewhere where it shouldn’t have and understandably logged :

Heads up - Critical Patch Update affecting OBIEE

Fri, 16 Oct 2009 00:00:00 +0000

The Critical Patch Update Pre-Release Announcement for October has been published. The pre-release is advance notice of the affected software prior to release of the quarterly Critical Patch Update. It is published on the Thursday prior to the patch releases (which was postponed by a week because of OOW).

It looks like if you’re running OBIEE 10.1.3.4.0 or 10.1.3.4.1 through OAS 10.1.2.3.0/10.1.3.4.0/10.1.3.5.0 then you should check back next Tuesday 20th for details.

New OBIEE benchmark - 50,000 users

Mon, 12 Oct 2009 00:00:00 +0000

A new OBIEE benchmark has been published by Oracle. It’s on the same hardware as August’s benchmark - Sun T5440s. Anyone would think that Oracle like Sun ;-)

Details here

Numbers added to my collation post here

OBIA 7.9.6.1 released

Mon, 12 Oct 2009 00:00:00 +0000

The latest point release of Oracle Business Intelligence Applications, 7.9.6.1, has been released and is available for download from here (direct link to download).

The version.txt reports the version as:

Build: 7.9.6.1.100609.2038 Release Version: Oracle Business Intelligence Applications 7.9.6.1 Package: 100609.2038

No updated documentation library yet through, so can’t nosey through the release notes. The docs that come with the download are labelled 7.9.6 and dated April 09 so don’t look like they’ve been updated either.

Usage Tracking - only half the story ...

Tue, 06 Oct 2009 00:00:00 +0000

OBIEE comes with a very useful usage tracking feature. For information about it and how to set it up see these links:

Usage Tracking captures the logical SQL of queries in a column called QUERY_TEXT in the table S_NQ_ACCT. However, out of the box this column is defined as 1k (1024 bytes) long. In some situations this will limit its usefulness because the text will be truncated if necessary when it’s inserted.

OBIEE and HP Performance Center (a.k.a. LoadRunner) - Notes

Thu, 01 Oct 2009 00:00:00 +0000

This is a supplemental post to this one describing how to set up a VUser in LoadRunner to test OBIEE. It’s various notes that I made during the development but which aren’t directly part of the step-by-step tutorial. They’re not necessarily vital for recording scripts, but observations and explanations that should be helpful when working with LoadRunner and OBIEE.

Validation using sawserver logs

It’s no use running a load test if the load you think you’re applying isn’t actually being applied. To validate the test you compare what happens on the server when the scenario is manually performed with what happens with it’s from a VUser and hopefully the same behaviour is observed.

Performance testing OBIEE using HP Performance Center (a.k.a. LoadRunner)

Thu, 01 Oct 2009 00:00:00 +0000

My two earlier posts (here and here) detail the difficulties I had with LoadRunner (now called HP Performance Center). After a bit of a break along with encouragement from knowing that it must be possible because it’s how Oracle generates their OBIEE benchmarks I’ve now got something working. I also got a useful doc from Oracle support which outlines pretty much what I’ve done here too.

In essence what you do - and what the Metalink document 496417.1 states - is you use the Web (HTTP/HTML) protocol with URL-mode.

James Morle : Spotting the Red Flags (Part 1 of n)

Fri, 25 Sep 2009 00:00:00 +0000

A new blog from James Morle, who I don’t know but from other bloggers sounds well respected, and describes himself thus:

Since it’s been nearly ten years since I wrote my book, Scaling Oracle8i, I thought it was about time that I started writing again. I thought I would start with the new-fangled blogging thing, and see where it takes me. Here goes.

He’s got a really interesting post on “red flags” to look for in diagnosing performance problems in Oracle: Spotting the Red Flags (Part 1 of n).

SQL Developer v2.1 Early Adopter released

Fri, 25 Sep 2009 00:00:00 +0000

SQL Developer v2.1 Early Adopter was released yesterday.

Download it here

New features list

Hat-tip and hat-tip

Changing password on Oracle 11g from 10g clients (ORA-28001 -> ORA-01017)

Wed, 23 Sep 2009 00:00:00 +0000

Bit of an odd one this. Oracle 11g database, a user’s password has expired. But when I try to change it, I can’t:

$sqlplus MYUSER/oldPW@oraDBServer

SQL*Plus: Release 10.2.0.1.0 - Production on Wed Sep 23 07:57:41 2009

Copyright (c) 1982, 2005, Oracle.  All rights reserved.

ERROR:
ORA-28001: the password has expired


Changing password for MYUSER
New password:
Retype new password:
ERROR:
ORA-01017: invalid username/password; logon denied


Password unchanged

After a bit of digging around I found a post (cached) which says that this is a problem when you use 10g clients with 11g database. And sure enough:

Collated OBIEE benchmarks

Fri, 18 Sep 2009 00:00:00 +0000

(Updated 12th Oct 09)

Here’s a list of the OBIEE benchmark documents published by Oracle:

Benchmark	Date	Source document
1 - IBM System x3755	Sep-07	PDF
2 - HP DL380 G4	Sep-07	PDF
3 - Sun T2000	Sep-07	PDF
4 - Sun SPARC Enterprise T5440	Aug-09	PDF
5 - Sun SPARC Enterprise T5440	Oct-09	PDF

Collecting the numbers into one table gives this:

Based on the details in the documents I think these were all against OBIA’s Service Analytics schema & dashboards/reports.

Interesting to note the side-by-side comparison in benchmark 3 (Sun T2000) of two servers, in one case both running BI and Presentation Services and in the other having the two components separate. It appears to highlight the benefit that clustering provides in making the best use of resources.

OBIEE benchmarks

Thu, 17 Sep 2009 00:00:00 +0000

Here’s a list of the OBIEE benchmark documents published by Oracle:

OBIEE cluster controller failover in action

Tue, 15 Sep 2009 00:00:00 +0000

Production cluster is 2x BI Server and 2x Presentation Services, with a BIG-IP F5 load balancer on the front.

Symptoms

Users started reporting slow login times to BI. Our monitoring tool (Openview) reported that “BIServer01 may be down. Failed to contact it using ping.”. BIServer01 cannot be reached by ping or ssh from Windows network.

Diagnostics

nqsserver and nqsclustercontroller on BIServer01 was logging these repeated errors:

[nQSError: 12002] Socket communication error at call=send: (Number=9) Bad file number

OBIEE on Opera 10 / IE 7 / FF 3.5 / Chrome 4

Thu, 10 Sep 2009 00:00:00 +0000

A new version of the web browser Opera was released recently. Several years ago I used Opera and may even have paid for it IIRC. Then Firefox came along, and the “it’s not IE” excuse was lost for Opera. Not that I mind IE too much nowadays but at the time it was atrocious. Since then I’ve revisited Opera each time a new release has come out, but nothing has impressed me enough to ditch Firefox (and nowadays Google Chrome).

Syntax for AdminTool.exe command line script

Wed, 09 Sep 2009 00:00:00 +0000

Bringing together in one place all of the script syntax that I’ve found so far for using with OBIEE’s AdminTool.exe /command

Details and examples on usage in the following blogs (where I compiled the commands from):

Venkat
Erik Eckhardt (translated from Czech, original here)
@lex
Kumar Kambam

DON’T TRY THIS AT HOME!

I would only recommend this for read-only purposes such as generating the metadata dictionary or consistency check.

* OpenOnline DSN [user [password]] - Opens the online repository. NB can’t edit properties without checking out objects first, and no way to do that from script.

AdminTool.exe /command

Tue, 08 Sep 2009 00:00:00 +0000

There’s an undocumented feature in AdminTool.exe that you can use the /command switch with a text file containing scripted commands to make changes to an RPD file (or create a new one).

It’s undocumented and UNSUPPORTED so be careful using it.

Some good details in these blog posts, especially Erik’s which has a good list of syntax.

Venkat
Erik Eckhardt (translated from Czech, original here)
@lex
Kumar Kambam

I’m intrigued to know how the original posters figured out the commands available, if it’s undocumented… :)

Metalink 3 followup

Fri, 04 Sep 2009 00:00:00 +0000

Kudos to the My Oracle Support blog for taking the time to respond to my to my comment about searching for Metalink 3 SRs throwing an error.

In essence, if you previously used Metalink 3 you must use https://support.oracle.com. If you use https://metalink.oracle.com/ then you’ll hit the problems I did.

ML3 SR’s are not supported on https://metalink.oracle.com/CSP/ui/index.html. This front end is used for support on legacy server technology, middleware including BEA, and EBusinessSuite

Full details here

RSS feeds for OBIEE, including del.icio.us obiee tags

Fri, 04 Sep 2009 00:00:00 +0000

Another way of keeping up with what’s going on in the obiee world, add this RSS feed of del.icio.us obiee tags to your reader. It may be less “current” (because people might discover and bookmark ‘old’ pages), but it’s another tool in the armoury :)

If you want an aggregated RSS feed of OBIEE / Oracle related blog postings try this one.

CAF troubles

Thu, 03 Sep 2009 00:00:00 +0000

Following the Oracle CAF tutorial here, I got to Cloning Answers Requests section and then got stuck. I’d set up my environment exactly the same as in the tutorial, down to the same paths etc. After firing up the CAF to clone requests from the SampleSales catalog: I clicked on Next and got the error “Exception occurred when while initializing repository!!!”

By playing around with the passwords and path names I determined that both RPD files existed and that CAF could load them enough to validate the passwords. If the password is incorrect you get the error “The repository C:\CAF_Training\Source\samplesales.rpd cannot be opened”.

Metalink 3 RIP

Wed, 02 Sep 2009 00:00:00 +0000

This weekend just gone Metalink3 went to the digital dustbin. In principle this is a Good Thing, as multiple support websites for a single company is confusing and frustrating.

Metalink is now “My Oracle Support” and is a flash-based whizz-bang affair. Everyone has different tastes, but there’s a lot to be said for plain HTML for ease and speed of access. But then people probably grumbled to the Wright Brothers that there was nothing wrong with land-transport at the time…

Multiple RPDs on one server - Part 1 - the BI Server

Tue, 25 Aug 2009 00:00:00 +0000

Introduction

In this article I plan to get samplesales and paint repositories hosted on a single server, using one BI Server instance and two Presentation Services instances. This is on both Unix (OEL 4) and Windows, and both OC4J (OBIEE’s “basic installation” option) and OAS (“Advanced Installation”).

Both samplesales and paint are shipped with 10.1.3.4 of OBIEE, you’ll find them in $OracleBI/OracleBI/server/Sample. This article assumes you’ve got the RPD of each into $OracleBI/OracleBI/server/Repository and unpacked the web cats for each into $OracleBIdata/web/catalog. It also assumes that you know your way around the architecture of BI and are familiar with NQSConfig.ini and instanceconfig.xml - if neither of those files mean anything to you then you will find some background reading useful.

Multiple RPDs on one server - Part 2 - Presentation Services

Tue, 25 Aug 2009 00:00:00 +0000

Introduction

In this article I plan to get sample and paint repositories hosted on a single server, using one BI Server instance and two Presentation Services instances. This is on both Unix (OEL 4) and Windows, and both OC4J (OBIEE’s “basic installation” option) and OAS (“Advanced Installation”).

Make sure you’ve read and followed part 1 - BI Server first.

Remember that multiple Presentation Services instances on a machine is UNSUPPORTED BY ORACLE!

OBIEE error/message code reference

Mon, 24 Aug 2009 00:00:00 +0000

For some reason Oracle haven’t put out a 10.x version of the error & message codes reference guide for OBIEE, but the previous version for Siebel Analytics is still useful:

PDF version: http://download.oracle.com/otndocs/products/bi/bi-ee/docs/784/AnyMsg.pdf
HTML version: http://download.oracle.com/docs/cd/E12103_01/books/AnyMsg/AnyMsg_Messages.html#wp1007961

Tech Support Cheat Sheet [xkcd.com]

Mon, 24 Aug 2009 00:00:00 +0000

xkcd.com is one of my favourite comics on the web. It strikes just the right balance of geeky witty humour without being too smart-ass.

I liked this recent one a lot :-)

A couple of other favourites:

OBIEE and Load Runner - part 2

Fri, 21 Aug 2009 00:00:00 +0000

UPDATED: See a HOWTO for OBIEE and LoadRunner here: /2009/10/01/obiee-and-loadrunner-howto/

This is following on from my first post about OBIEE and LoadRunner, in which I failed dismally to get a simple session replaying.

In a nutshell where I’d got to was using the “Web (Click and Script)” function which worked fine for logging in but when running a report resulted in an error on the rendered page. Digging around showed the error was from the javascript of the OBIEE front end.

Querying SQL Server from OBIEE running on Unix

Fri, 21 Aug 2009 00:00:00 +0000

A question that pops up on the OBIEE OTN forum quite often is how to use non-Oracle databases like MS SQL Server when the OBIEE server is running on a non-Windows OS such as Linux.

The answer in a nutshell is that since version 10.1.3.3.1 OBIEE has been bundled with ODBC drivers for unix/linux from a company called DataDirect. See the release notes here for more information and installation instructions (as well as a list of support databases).

Do you mean (pt II)

Thu, 20 Aug 2009 00:00:00 +0000

A follow up to my previous post about Metalink’s “Do you mean” feature, this one made me laugh:

I shall miss this kind of thing when Metalink3 merges into My Oracle Support….

Meep meep!

Logging specific types of sawserver activity

Thu, 20 Aug 2009 00:00:00 +0000

As well as tinkering with the sawserver (Presentation Services) logging level and format, we can specific which bits of the log we’re interested in. This is useful for two reasons:

We can enable detailed logging for a specific area, without impacting performance as much as detailed logging throughout would cause
By only logging in detail the area of interest we can more easily read the log output and not have to wade through pages of irrelevant information

Chapter 9 (“Using the Oracle BI Presentation Services Logging Facility”) of the Presentation Services Administration Guide details the log configuration.

OBIEE and Load Runner - part 1

Wed, 19 Aug 2009 00:00:00 +0000

UPDATED: See a HOWTO for OBIEE and LoadRunner here

Introduction

LoadRunner is a tool from HP (bought from Mercury) that can be used to simulate user activity. It supports a whole host of protocols but for OBIEE I’m obviously using the Web one.

There are two flavours, “Web (Click and Script)” and “Web (HTTP/HTML)”. The latter simply shoves HTTP requests at the server, whereas “Click and Script” simulates mouse and keyboard entry and thus is more appropriate for this user-based application. [edit]I’m not sure if this is actually the case[/edit]

sawserver log - short format

Wed, 19 Aug 2009 00:00:00 +0000

I posted a while ago about the sawserver (Presentation Services) log configuration file. Today I’m doing some work digging around why sawserver’s throwing an error and so increased the log detail. This parameter is really helpful to use:

fmtName=“short”

Consider in these two screenshots, the first is with the default log format and shows about six entries. The second is short log format and is about ten times as much data.

OTN forums - different URL to get prompted to login less often

Tue, 18 Aug 2009 00:00:00 +0000

I have a couple of OTN forums bookmarked, and found that generally every few hours I get signed out and end up viewing them as Guest. I then have to click on sign in, enter password, etc.

(Signing out this frequently is ridiculous, IMHO).

I found that instead of using the direct URL of a forum, eg: http://forums.oracle.com/forums/forum.jspa?forumID=378 if I use this form: http://forums.oracle.com/forums/adfAuthentication?success_url=https://rmoff.net/forum.jspa?forumID=378 then I end up signed in more often, and if I’ve been signed out then it goes straight to the login page and then through to the forum.

Unix script to report on OBIEE and OBIA processes state

Fri, 14 Aug 2009 00:00:00 +0000

Here’s a set of scripts that I use on our servers as a quick way to check if the various BI components are up and running.

Because we split the stack across servers, there are different scripts called in combination. On our dev boxes we have everything and so the script calls all three sub-scripts, whereas on Production each server will run one of:

BI Server
Presentation Server & OAS
Informatica & DAC

The scripts source another script called process_check.sh which I based on the common.sh script that comes with OBIEE.

OBIA upgrade 7.9.5 to 7.9.6 - first thoughts

Thu, 13 Aug 2009 00:00:00 +0000

We’re upgrading from OBIA 7.9.5 (Financials - GL) to OBIA 7.9.6. Our reasons are for support (7.9.5 does not support Oracle 11g) and minor functionality additions.

Our architecture is: HP-UX 64 bit Itanium (11.31), Oracle 11g (11.1.0.7), separate ETL server, 4x OBIEE servers (2x BI, 2xPS). We have no customisations in the ETL except something for budgets, which is superseded in 7.9.6.

This post is a semi-formed articulation of my frustrations encountered during an initial run through of the upgrade in a sandbox. As we progress with the upgrade I will post further, hopefully more useful, information on what we encounter.

Repository Error ([REP_51821] Failed to connect from Integration Service (pmserver) to repository Oracle_BI_DW_Base running in exclusive mode.)

Mon, 10 Aug 2009 00:00:00 +0000

I keep hitting this error when setting up OBIA. I suppose it’s what it says on the tin, but Googling it didn’t match so I’m posting this so next time I hit it I remember :-)

Repository Error ([REP_51821] Failed to connect from Integration Service (pmserver) to repository Oracle_BI_DW_Base running in exclusive mode.)

The cause is the Repository Service having OperatingMode set to Exclusive. This is necessary for some of the setup operations like restoring the pre-built Repository, but if you forget to switch it back the Integration Service will suddenly stop working.

Clean install of OAS - Enterprise Manager not available

Thu, 06 Aug 2009 00:00:00 +0000

I successfully installed OAS 10.1.3.3 and patched to 10.1.3.4. http://localhost:7777 gave the OAS welcome page, but going to http://localhost:7777/em gave 404 Not Found.

In [OASHome]/j2ee/home/config/servers.xml search for ascontrol, you should get:

change the start attribute to true

Restart OAS ([OAShome]/opmn/bin/opmnctl restartproc) and Enterprise Manager should now be available

Google vs Bing

Wed, 05 Aug 2009 00:00:00 +0000

There’s been a bit of hype about Bing recently, so I thought I’d try it out in trying to get to the bottom of this question on the OBIEE forum.

The question was around the nqschangepassword utility and the error it’s reporting: nQSError: 46090 The odbc.ini file could not found or could not be accessed.

I did a google for the error to see what other issues could cause the error. Google showed up the forum posting in question, plus the Siebel 794 message reference PDF (no v10 reference guide yet :( ) (in blogging this I’ll now probably show up on the list too!)

OBIA grumble

Tue, 04 Aug 2009 00:00:00 +0000

I’m starting on an upgrade from OBIA 7.9.5 to 7.9.6 and wading through the two main docs:

It would be nice if Oracle could come up with some less confusing terminology. It seems that not only is the whole product of OBIA referred to as OBIA (see @lex’s posting for a good explanation), but that the sub-components which are not-OBIEE-or-DAC-or-Informatica is also OBIA, c.f. page 6-1 of the Upgrade guide “[…]upgrade your Oracle BI Applications environment to the current version.” To me that implies that once I’ve done this, my OBIA will be upgraded - but no, actually, some of the supporting bits will be upgraded, but I still have to do a heck of a lot more grunt work before what I consider OBIA (and the manual is called OBIA too!) is upgraded.

What is OBIA...

Thu, 30 Jul 2009 00:00:00 +0000

Very good post by @lex giving a nice clear explanation of what OBIA (Oracle Business Intelligence Applications) is

http://siebel-essentials.blogspot.com/2009/06/can-you-describe-oracle-bi-applications.html

This should be made a sticky on the OBIA forum in my opinion.

It’s clear from postings on the forum that an awful lot of people don’t understand what OBIA is or how it sits with OBIEE. I even attended a course last week in which the Oracle trainer stated that OBIEE included ETL and DW schemas, and stuck to this when challenged.

OBIEE performance monitoring and alerting with jManage

Wed, 29 Jul 2009 00:00:00 +0000

OBIEE’s Systems Management component exposes configuration and performance data through Java MBeans. As discussed in other posts these can be be accessed through several different ways:

JConsole (see also here)
oc4j
Windows PerfMon (although I guess this isn’t actually using MBeans/JMX?)
saw.dll?perfmon
BI Management Pack

Since it’s a standard java technology being used we can in theory use anything that is designed for monitoring mbeans via jmx. Doing some Googling I discovered jManage.

How to find out what web application server is in use

Tue, 28 Jul 2009 00:00:00 +0000

If, for some reason, you need to check what web application server is in use for Presentation Services (as this chap needed to), you can use an add-in for FireFox called HttpFox to inspect the HTTP headers.

1. Install HttpFox (and obviously Firefox if you don’t have it already!)
2. Open the HttpFox window (Tools -> HttpFox -> Toggle HttpFox)
3. Click the Start button in the HttpFox window
4. Navigate to your OBIEE home page
5. Click the Stop button in the HttpFox window
6. Click on the first entry in the list, URL should be http://yourserver:7777/analytics/saw.dll?Dashboard
7. In the right-hand pane of the Headers tab you should see Server listed. In this instance, it’s Oracle-Application-Server-10g/10.1.3.1.0 Oracle-HTTP-Server

Maker's Schedule, Manager's Schedule

Tue, 28 Jul 2009 00:00:00 +0000

I found this post very interesting: Paul Graham : Maker’s Schedule, Manager’s Schedule (originally found here)

It was one of those mini-revelations when I found something that greatly resonated and explained an inexplicable frustration I find in the workplace sometimes.

I wonder how applicable it can be to a large corporation though, rather than a start-up where it’s given that people are allowed to be bolshy with their meetings? ;-)

ORA-00922: missing or invalid option

Mon, 27 Jul 2009 00:00:00 +0000

We routinely change Oracle passwords as part of security best-practice, I keep hitting this and keep forgetting why! :-)

ALTER USER DAC_REPO IDENTIFIED BY 1KoBe3RH REPLACE YlR94tqp

Error report: SQL Error: ORA-00922: missing or invalid option 00922. 00000 - “missing or invalid option” *Cause: *Action:

Someone better qualified than me can explain why but I suspect it’s the leading number in the new password. Quoting the passwords then works fine:

Metalink 3 - Do You Mean ... ?

Fri, 24 Jul 2009 00:00:00 +0000

One of my little gripes with Metalink is its purporting to be helpful when it’s blatantly not. Here’s one:

Now which is more likely, on Metalink 3; that I’m searching for sawserver (integral component to OBIEE), or sqlserver?!

OBIEE Windows PerfMon counters

Fri, 24 Jul 2009 00:00:00 +0000

Yet another way to access the BI Management data discussed here - through Windows’ PerfMon tool.

This will only work for installations where your OBIEE server is running on Windows. You should be able to run PerfMon locally or remotely. Standard practise would be not to run it locally on a Production machine :-)

To run PerfMon go to Start->Run and enter perfmon, or navigate Start -> Settings -> Control Panel -> Administrative Tools -> Performance

Mark Rittman's OBIEE repository for DAC

Thu, 23 Jul 2009 00:00:00 +0000

Mark Rittman has an excellent article about querying the DAC repository database tables, including a downloadable RPD file. Being new to working with RPDs I thought it would be good practise to explore this as well as hopefully get some useful information about our current ETL deployment.

I downloaded the RPD to c:\OracleBI\server\Repository and opened it up in the Admin tool (Administrator/Administrator).
First off I changed the connection pool to point to my DAC repository database, having setup a TNS entry for it first.

psservice - Windows command line goodness!

Thu, 23 Jul 2009 00:00:00 +0000

Our main servers are Unix and I’m as happy as a pig in muck at the command line, so when I’m working on Windows (where I’ve got a test OBIEE install) I like to stick with the CLI where possible.

PSService is one of those tools that I instinctively reach for without realising it. Combined with Launchy, it’s even better.

Simply put, you can control windows services from the command line.

sawserver charts crash

Thu, 23 Jul 2009 00:00:00 +0000

By a strange co-incidence after following this thread on OTN forums about a BI crash and struggling to understand the actual problem, I think I’ve encountered it myself!

I’ve got a test install of OBIEE running on my Windows XP laptop, and whilst building a report in Answers got this:

](http://2.bp.blogspot.com/_RCx_EVJpczQ/SmiDrEorbbI/AAAAAAAAGcg/OPGwTLXXg8k/s1600/crash1.png)[![](/images/rnm1978/crash2.png) was:

szAppName : sawserver.exe szAppVer : 10.1.3.4 szModName : kernel32.dll szModVer : 5.1.2600.3119 offset : 000097a3

Going to the sawserver log at c:\OracleBIData\web\log\sawlog0.log disappointingly showed no error entries :(

sawserver logging configuration - logconfig.xml

Thu, 23 Jul 2009 00:00:00 +0000

The configuration of how Presentation Services (sawserver) does its logging is in the file web/config/logconfig.xml (same directory as instanceconfig.xml).

It’s all nice and XML’d:

Logging Detail
Change the numerical values in the FilterRecord entries to alter the detail level of the logging. Lower means less detail, higher means more.

Be aware that your log files can grow very rapidly if you set the logging too high, and unless you’re troubleshooting then leave them at the defaults.

Oracle BI Management / Systems Management MBeans

Wed, 22 Jul 2009 00:00:00 +0000

Part of looking at the various gubbins inside OBIEE led me to realise that the Oracle BI Management application drives quite a few things. It exposes MBeans (Management Beans, a java term), accessible through jmx. In the installation of OBIEE this component is referred to as “Systems Management”.

The MBeans give us real-time performance information, along with access to all the configuration options that are normally done through config files (instanceconfig.xml etc). Bear in mind if using it for updating configuration instead of through the files you don’t get any backup created, so for that reason alone I would suggest it should only be used for reading current values.

JConsole / JMX - followup

Tue, 21 Jul 2009 00:00:00 +0000

A few points to add to my previous posting on JConsole:

As well as performance data, you have access to configuration data. Be aware that it is read-write! So whilst it might be a nice alternative to digging around for your instanceconfig.xml etc, you should be careful
If you have your BI Server and Presentation Services deployed on separate servers then you will only get MBeans for the relevant service:

OBIEE admin tools & hacks

Tue, 21 Jul 2009 00:00:00 +0000

As a kid I loved the idea of lego where you can disassemble and reassemble something from the ground up. As soon as I got my hands on a computer it was the same. You can have your Acorn Archimedes with its games, where do I find the sprites and sound files behind it? Likewise Microsoft Word, let me at the VBA underneath to hack it around and see what else it can do.

OTN forum rant

Tue, 21 Jul 2009 00:00:00 +0000

I read and post a bit on the OBIEE and OBIA OTN forums. The noise ratio isn’t too bad, but a few things really get my goat:

Not responding to answers!
If I’ve gone out of my way to help, or try to help, at least have the courtesy to acknowledge it, and ideally mark as Helpful or Correct as appropriate.
Even a simple “thanks.” would do. It’s just good manners.
It also helps people who come along afterwards as you get the completed picture rather than half a question/answer thread.
Forums Etiquette

JConsole / JMX

Thu, 16 Jul 2009 00:00:00 +0000

[edit] See this post too [/edit] On an OBIEE server run

nohup obiee/systemsmanagement/runagent.sh &amp;

and then run jconsole (make sure you’ve set the DISPLAY first if you’re running it from UNIX). NB: if you don’t have jconsole in your path you can search for it:

$whereis jconsole
jconsole: /opt/java1.5/bin/jconsole /opt/java6/bin/jconsole&lt;/span&gt;

You should find it under your java/bin directory

You should get this kind of connection dialog: Click connect, and the console will launch. From here click on the MBeans tab, where you’ve got access to performance and configuration data

nqcmd and [nQSError: 27005] Unresolved column

Thu, 28 May 2009 00:00:00 +0000

I’m working on a scripted load test for OBIEE using nqcmd to run reports multiple times. I hit this interesting issue.

Cut and pasting the logical SQL that Presentation Services sends to BI Server from Manage Sessions -> Statement, I kept getting this error when I ran it through nqcmd:

[10058][State: S1000] [NQODBC] [SQL_STATE: S1000] [nQSError: 10058] A general error has occurred.
[nQSError: 27005] Unresolved column: “Natural Account (COA)”.“Account Parent1 Code”.
Statement preparation failed

Custom HTTP error page in OBIEE / OAS

Mon, 18 May 2009 00:00:00 +0000

It’s possible to change the error pages served up by OAS/Apache by using the ErrorDocument directive. This is widely documented.

However, to get this to take effect in an oc4j application (such as analytics) you need to change mod_oc4j.conf too.
(I found this out from this post here)

Take backups of httpd.conf and mod_oc4j.conf, and then edit them as follows:

In httpd.conf add:
ErrorDocument 500 /500.html
where /500.html is a relative path to your custom document

New releases

Tue, 28 Apr 2009 00:00:00 +0000

New releases this week - 10.1.3.4.1 of OBIEE, and 7.9.6 of OBIA.

After a bit of scrabbling around found:

“The 10.1.3.4.1 release of the Oracle Business Intelligence Enterprise Edition introduces no new features.”

Though there are some new bits and pieces for Publisher

For 7.9.6 OBIA, there’s no New Features document :(
By piecing together the “What’s New in This Release” for each document you can build up a picture (eg in the Upgrade Guide there’s reference to changes for the doc relating to a new version of Informatica), but it would be nice to have it all in one place:

OBIEE and F5 BIG-IP

Wed, 15 Apr 2009 00:00:00 +0000

We’ve got a setup of two OAS/Presentation Services boxes and two BI Server boxes, with load balancing/failover throughout.
The Load Balancing of the web requests is being done through a separate bit of kit, an F5 BIG-IP load balancer. This directs the requests at the two OAS servers.

The problem we have is that by default OAS serves HTTP on port 7777, but the F5 is using port 80. A request for our load balanced URL: http://bi.mycompany.com/analytics/ barfs out with

Google and Korean OTN forums

Thu, 02 Apr 2009 00:00:00 +0000

Why does Google more often than not return the korean (kr) version of OTN forums for a match, but not english?

If I search Google for “Send notification:” “oracle.ons.Notification” the top hit is for kr.forums.oracle.com/forums/thread.jspa?threadID=650662

If you strip the kr from the URL you get this, which makes more sense given the locality when doing the search in Google.

Only mildly irritating, but odd behaviour nonetheless

OAS bug 7132128 - Send notification: oracle.ons.Notification

Thu, 02 Apr 2009 00:00:00 +0000

I noticed that the j2ee server.log file was filling up with these entries:

oracle.ons.Notification@afba5d
09/04/02 10:15:12.207 Send notification:
oracle.ons.Notification@17ca0f5
09/04/02 10:15:42.217 Send notification:
oracle.ons.Notification@1a28842
09/04/02 10:16:12.227 Send notification:
oracle.ons.Notification@5144d5
09/04/02 10:16:42.237 Send notification:
oracle.ons.Notification@19078ed
09/04/02 10:17:12.247 Send notification:
oracle.ons.Notification@fcc268
09/04/02 10:17:42.257 Send notification:
oracle.ons.Notification@16df388

A quick google turned up this page in which bug 7132128 (“OC4J EMITS “SEND NOTIFICATION” MESSAGES THAT FILL SERVER.LOG”) is identified.

The server.log is currently only 6MB and given the size of the server it will be a while before it causes us problems, but it’s worth being aware of.

OAS makes you log in twice

Thu, 02 Apr 2009 00:00:00 +0000

A very minor irritation, but an irritation nonetheless, is when I go to Application Server Control in OAS I have to login twice.

Reading around I found this is an Apache feature, and is actually designed behaviour.

For reasons I’ve not explored our servers have several different hostnames which resolve to the same IP, e.g.:
myserver
myserver-app
myserver-data

When you request a page from Apache using a hostname other than that configured as ServerName in Apache’s httpd.conf, it redirects you to the version of the page using the ServerName.

sawserver won't start up - resolved

Wed, 01 Apr 2009 00:00:00 +0000

(See here and here for history)

I edited the shell script which is eventually called by run-saw.sh to start the sawserver, (OracleBI)/setup/sawserver.sh, to use trus:
Comment out the final line:

$SASAWSERVER

and insert as a new line:

tusc -fepan -o /tmp/sawserver_tusc.out $SASAWSERVER

The output of trus ended with this:

open("/app/oracle/product/10.2.0/lib/libstd_v2.so.1", O_RDONLY|0x800, 0) ……… ERR#2 ENOENT
open("/app/oracle/product/obiee/server/Bin64/libstd_v2.so.1", O_RDONLY|0x800, 0) . ERR#2 ENOENT
open("/app/oracle/product/obiee/web/bin64/libstd_v2.so.1", O_RDONLY|0x800, 0) …. ERR#2 ENOENT
open("/app/oracle/product/obiee/odbc/lib64/libstd_v2.so.1", O_RDONLY|0x800, 0) … ERR#2 ENOENT
open("/usr/lib/libstd_v2.so.1", O_RDONLY|0x800, 0) …………………………. ERR#2 ENOENT
open("/lib/libstd_v2.so.1", O_RDONLY|0x800, 0) …………………………….. ERR#2 ENOENT
open("/app/oracle/product/10.2.0/lib/libstd_v2.so.1", O_RDONLY|0x800, 0) ……… ERR#2 ENOENT
open("/opt/aCC/lib/hpux64/libstd_v2.so.1", O_RDONLY|0x800, 0) ……………….. ERR#2 ENOENT

Troubleshooting an HPUX program

Wed, 01 Apr 2009 00:00:00 +0000

In investigating the problems with sawserver I was pointed towards a tool called tusc (which appears to be an HP version of truss).

You can use it to invoke a program, and get out a bunch of debug information including system calls.

You run it like this:

$tusc -fep /app/oracle/product/obiee/web/bin64/sawserver64

As a beginner when it comes to hardcore *nix I can only look at this and take pot shots at what’s going on, but with Google by my side I’m interested in the last lines of the output:

Bug in Clustered Publisher Scheduler - ClusterManager: detected 1 failed or restarted instances

Mon, 30 Mar 2009 00:00:00 +0000

Follow on from setting up Publisher in a clustered environment, I’ve found a nasty little bug in the scheduling element of Publisher, Quartz.

Looking at the oc4j log file /opmn/logs/default_group~~home~~default_group~1.log I can see OC4J starting up, and then a whole load of repeated messages:

09/03/30 11:28:43 Oracle Containers for J2EE 10g (10.1.3.3.0) initialized
- ClusterManager: detected 1 failed or restarted instances.
- ClusterManager: Scanning for instance “myserver.fqdn.company.net1238408921404”’s failed in-progress jobs.
- ClusterManager: detected 1 failed or restarted instances.
- ClusterManager: Scanning for instance “myserver.fqdn.company.net1238408921404”’s failed in-progress jobs.
- ClusterManager: detected 1 failed or restarted instances.
- ClusterManager: Scanning for instance “myserver.fqdn.company.net1238408921404”’s failed in-progress jobs.
- ClusterManager: detected 1 failed or restarted instances.
- ClusterManager: Scanning for instance “myserver.fqdn.company.net1238408921404”’s failed in-progress jobs.
- ClusterManager: detected 1 failed or restarted instances.
- ClusterManager: Scanning for instance “myserver.fqdn.company.net1238408921404”’s failed in-progress jobs.
- ClusterManager: detected 1 failed or restarted instances.
- ClusterManager: Scanning for instance “myserver.fqdn.company.net1238408921404”’s failed in-progress jobs.
- ClusterManager: detected 1 failed or restarted instances.
- ClusterManager: Scanning for instance “myserver.fqdn.company.net1238408921404”’s failed in-progress jobs.
- ClusterManager: detected 1 failed or restarted instances.
- ClusterManager: Scanning for instance “myserver.fqdn.company.net1238408921404”’s failed in-progress jobs.
[… repeated for 38MB worth ]

sawserver won't start (analytics: Servlet error java.net.ConnectException: Connection refused (errno:239))

Mon, 30 Mar 2009 00:00:00 +0000

We’re getting this error in the Presentation Services plug-in [analytics].
Log file: /j2ee/home/application-deployments/analytics/home_default_group_1/application.log

09/03/30 13:16:38.75 analytics: Servlet error
java.net.ConnectException: Connection refused (errno:239)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:517)
at java.net.Socket.connect(Socket.java:467)
at java.net.Socket.(Socket.java:364)
at java.net.Socket.(Socket.java:178)
at com.siebel.analytics.web.sawconnect.ConnectionPoolSocketFactoryImpl.createSocket(ConnectionPoolSocketFactoryImpl.java:63)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at com.siebel.analytics.web.sawconnect.ConnectionPoolSocketFactoryImpl.createSocket(ConnectionPoolSocketFactoryImpl.java:70)
at com.siebel.analytics.web.sawconnect.ConnectionPool.createNewConnection(ConnectionPool.java:314)
at com.siebel.analytics.web.sawconnect.ConnectionPool.getConnection(ConnectionPool.java:133)
at com.siebel.analytics.web.SAWBridge.processRequest(SAWBridge.java:299)
at com.siebel.analytics.web.SAWBridge.doGet(SAWBridge.java:325)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:743)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.3.0) ].server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:713)
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.3.0) ].server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:3
70)
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.3.0) ].server.http.HttpRequestHandler.doProcessRequest(HttpRequestHandler.java:871)
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.3.0) ].server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:453)
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.3.0) ].server.http.AJPRequestHandler.run(AJPRequestHandler.java:302)
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.3.0) ].server.http.AJPRequestHandler.run(AJPRequestHandler.java:190)
at oracle.oc4j.network.ServerSocketReadHandler$SafeRunnable.run(ServerSocketReadHandler.java:260)
at oracle.oc4j.network.ServerSocketAcceptHandler.procClientSocket(ServerSocketAcceptHandler.java:239)
at oracle.oc4j.network.ServerSocketAcceptHandler.access$700(ServerSocketAcceptHandler.java:34)
at oracle.oc4j.network.ServerSocketAcceptHandler$AcceptHandlerHorse.run(ServerSocketAcceptHandler.java:880)
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.3.0) ].util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.
java:303)
at java.lang.Thread.run(Thread.java:595)

ODI Server install - missing odiparams.sh file

Fri, 27 Mar 2009 00:00:00 +0000

I’m installing ODI agent on our database server using OUI. I selected the “Server” option at install time to get the Agent only, but looking in oracledi/bin odiparams.sh is missing:

$ls -l *.sh
-rwxrwxrwx 1 odiadm dba 685 Nov 21 15:58 agent.sh
-rwxrwxrwx 1 odiadm dba 908 Nov 21 15:58 agentscheduler.sh
-rwxrwxrwx 1 odiadm dba 707 Nov 21 15:58 agentstop.sh
-rwxrwxrwx 1 odiadm dba 941 Nov 21 15:58 agentweb.sh
-rwxrwxrwx 1 odiadm dba 724 Nov 21 15:58 jython.sh

Remove windows line feed characters in vi

Fri, 27 Mar 2009 00:00:00 +0000

If you work with a file in Windows and Unix at some point you might end up with windows line feed characters in your Unix file. It’ll look like this:

one line of text ^M
next line ^M
and next line with more ^M

To remove the ^M character, load the file into vi on unix and enter as a line command the following:

:1,$s/^M//

but instead of typing ^M do Ctrl-V Ctrl-M to get the charaters

ORA-12537 / ORA-12518 [Informatica DAC error CMN_1022]

Wed, 25 Mar 2009 00:00:00 +0000

We’re getting problems with an instance of Informatica / out-of-the-box OBIA on a new set of servers. When we run the execution plan we get this error soon after starting:

MAPPING> DBG_21075 Connecting to database [TNSENTRY], user [MYUSER]
MAPPING> CMN_1761 Timestamp Event: [Tue Mar 24 18:56:33 2009]
MAPPING> CMN_1022 Database driver error…
CMN_1022 [
Database driver error…
Function Name : Logon
ORA-12537: TNS:connection closed

Database driver error…
Function Name : Connect
Database Error: Failed to connect to database using user [MYUSER] and connection string [TNSENTRY].]

Clustering Publisher - Scheduler and Report Repository

Tue, 24 Mar 2009 00:00:00 +0000

The Oracle BI Publisher Enterprise Cluster Deployment doc which I just found through Metalink highlighted a couple of points:
- Report repository should be shared
- The scheduler should be configured for a cluster

Report Repository
Through Admin>System Maintenance>Report Repository I changed the path from the default, /xmlp/XMLP to a NFS mount data/shared/xmlp and restarted the xmlpserver application in OAS. On coming back up Publisher complained because all its config files (in xmlp/Admin), had disappeared. I’d not moved any of the contents of /xmlp/XMLP since Report Repository suggested to me that it was just for reports, ergo with no reports yet created there was nothing to move.
So pedantaries aside, I moved the contents of /xmlp/XMLP to my new share, data/shared/xmlp. Publisher was happy after this.

Firefox add-ins - ones I find useful

Tue, 24 Mar 2009 00:00:00 +0000

Delicious Bookmarks
FireGestures
FoxyProxy
Greasemonkey
HttpFox
Screengrab
User Agent Switcher

Metalink Metalink Metalink

Tue, 24 Mar 2009 00:00:00 +0000

I’m learning about a lot of the Oracle BI stack through reading manuals and trial-and-error.
One thing I’ve realised is that Metalink holds a whole heap of useful information.
For example, simply searching on “publisher cluster” throws up these two very pertinent docs:

OBIEE Clustered Installation with BI Publisher (Doc ID 744515.1)
BI Publisher does not accept cluster jdbc connection strings (Doc ID 559795.1)

The first one is a publically available PDF, the second one is the answer to the problem I spent more time than I needed to scratching my head over yesterday.

Which jdbc driver to use

Tue, 24 Mar 2009 00:00:00 +0000

In setting the scheduler in Publisher I discovered a useful difference in jdbc drivers.
Our repository is on Oracle 11g.
According to the manual oracle.jdbc.driver.OracleDriver should be used, but previous installations have used oracle.bi.jdbc.AnaJdbcDriver so I tried this too.

In experimenting with both I found you get more useful feedback from the second one. Here’s the same problem reported by both drivers:

· Exception [TOPLINK-4002] (Oracle TopLink - 11g Release 1 (11.1.1.0.0) (Build 080319)): oracle.toplink.exceptions.DatabaseException Internal Exception: java.sql.SQLException: ORA-28000: the account is locked Error Code: 28000

Finding config files in unix

Mon, 23 Mar 2009 00:00:00 +0000

Following my previous work on configuring Publisher, I wanted to note down where the changes were written to.

The -mname syntax of the unix find command comes in handy here:

find /app/oracle/product/obiee -mtime -1

Shows me all files under the specified path which were modified in the last 1 day

and helpfully throws up:

/app/oracle/product/obiee/xmlp/XMLP/Admin/DataSource/datasources.xml

OBIEE Publisher - configuring connection to clustered BI Server

Mon, 23 Mar 2009 00:00:00 +0000

I’m setting up a clustered OBIEE 10.1.3.4 production environment. There are four servers; two BI Server + Cluster Controller + Scheduler and two OAS + Presentation Services + Publisher. Clustering of BI is configured, now I’m setting up the other bits. Today is Publisher.

On publisher instance A connections to the BI Servers directly work fine:
jdbc:oraclebi://serverA.fqdn.company.net:9703/ jdbc:oraclebi://serverB.fqdn.company.net:9703/
both work individually as Connection Strings (with database driver class of oracle.bi.jdbc.AnaJdbcDriver) - verified with “Test Connection” button.
Connections also work when specifying the hostname only (i.e. no FQDN).

About Me

Mon, 01 Jan 0001 00:00:00 +0000

`whoami`

Robin is a Sr. Principal Advisor, Streaming Data Technologies at Confluent.

He has been speaking at conferences since 2009 including QCon, Devoxx, Strata, Kafka Summit, and Øredev.

You can find his talks online, subscribe to his YouTube channel, and read his blog. Outside of work, Robin enjoys running, drinking good beer, and eating fried breakfasts—although generally not at the same time.

👉🏻 Bio & speaker photos

Speaking experience

Speaker since 2009 at conferences including QCon, Devoxx, O’Reilly Strata, NDC, USENIX LISA, Kafka Summit, Øredev, O’Reilly SACon, Oracle OpenWorld, JavaZone, Big Data LDN, UKOUG, Oracle CODE, PGConf, etc plus numerous meetups.

Search

Mon, 01 Jan 0001 00:00:00 +0000

Talks

Mon, 01 Jan 0001 00:00:00 +0000

See talks

Talks

Mon, 01 Jan 0001 00:00:00 +0000

You can find all my recent talks on Notist at https://talks.rmoff.net/. Older ones are on Speaker Deck.

I have a variety of talks, around several different aspects of Apache Kafka and related technologies.

Kafka 101 / introductory talk : “Kafka as a Platform: the Ecosystem from the Ground Up”
Introduction to ksqlDB and stream processing principles and semantics - with lots of live demos. “An introduction to ksqlDB”
Live demo showing use of Kafka Connect and ksqlDB: “Apache Kafka in Action : Let’s Build a Streaming Data Pipeline!”
Architectural view of Kafka and its implications for data engineering & data warehousing: “The Changing Face of ETL: Event-Driven Architectures for Data Engineers”
Deep-dive on Kafka Connect: “From Zero to Hero with Kafka Connect”
Nitty-gritty examining options of how you get data into Kafka from RDBMS “No More Silos: Integrating Databases and Apache Kafka”
PoC showing how to build streaming ETL/data platform around live data feeds: “🚂On Track with Apache Kafka: Building a Streaming Platform solution with Rail Data”
Example of integrating IoT data over MQTT into Kafka and processing it: “Building IoT applications with Kafka and ksqlDB”

Detailed abstracts are available on request for these. Recordings and slides for many can be seen at https://talks.rmoff.net/