DuckDB added a very cool UI last week and I’ve been using it as my primary interface to DuckDB since.
One thing that bothered me was that the SQL I was writing in the notebooks wasn’t exportable. Since DuckDB uses DuckDB in the background for storing notebooks, getting the SQL out is easy enough.
I wrote a couple of weeks ago about using DuckDB and Rill Data to explore a new data source that I’m working with. I wanted to understand the data’s structure and distribution of values, as well as how different entities related. This week DuckDB 1.2.1 was released and that little 0.0.1 version boost brought with it the DuckDB UI.
Here I’ll go through the same process as I did before, and see how much of what I was doing can be done in DuckDB alone now.
Much as I love kcat (🤫 it’ll always be kafkacat to me…), this morning I nearly fell out with it 👇
😖 I thought I was going stir crazy, after listing topics on a broker and seeing topics from a different broker.
😵 WTF 😵
Some would say that the perfect blog article takes the reader on a journey on in which the development process looks like this:
The UK Government publishes a lot of its data as open feeds. One that I keep coming back to is the Environment Agency’s flood-monitoring API that gives access to an estate of sensors that provide information about data such as river levels and rainfall.
The data is well-structured and provided across three primary API endpoints. In this blog article I’m going to show you how I use Flink SQL to explore and wrangle these into the kind of form from which I am then going to build a streaming pipeline using them.
There was a useful question on the Apache Flink Slack recently about joining data in Flink SQL:
How can I join two streams of data by id in Flink, to get a combined view of the latest data?
Let’s imagine we’ve got a source of data with a nested array of multiple values. The data is from an IoT device. Each device has multiple sensors, each sensor provides a reading.
The UK Environment Agency publishes a feed of data relating to rainfall and river levels. As a prelude to building a streaming pipeline with this data, I wanted to understand the model of it first.
I was exploring some new data, joining across multiple tables, and doing a simple SELECT * as I’d not worked out yet which columns I actually wanted.
The issue was, the same field name existing in more than one table.
This meant that in the results from the query, it wasn’t clear which field came from which table: