Interesting links - June 2026

🔥 Mina Tafreshi - Kafka Rebalances: What’s Actually Happening Under the Hood.

🔥 Michał Matłoka has been busy this month, first creating a tool that converts Kafka ACLs to Confluent RBAC, and then writing about a simulator for understanding how Kafka works. You can try the Kafka Simulator for yourself in your web browser. Related to this, he’s also written an article addressing the question Are Kafka Replicas Synchronous or Asynchronous?

OpenData’s Jason Gustafson and Almog Gavra (both Confluent-alumni) reckon there’s a better way to implement one of the two primary use cases for Kafka that they call 'routing', and they’re releasing OpenData Log to do it.

A trio of posts from Jack Vanlightly looking at Kafka Share Groups and beyond:

On the subject of share groups in Kafka, Sage Pierce has an alternative proposal, in the form of Atleon.

Another visualiser for Kafka, this one from Sandon Jacobs is specifically for KIP-932 Share Groups.

Trivago’s ZhongLi Shen details How We Cut Kafka Consumer Deployment Costs by 83%.

Ângelo Galvão shows how to Create a simple Mutual TLS (mTLS) authentication for Strimzi.

Sam Barker puts Kroxylicious through its paces, and shares a benchmark harness and results.

Elad Eldor - The Cheapest Kafka Consumer Is One That Doesn’t Read From Kafka.

Aswin A writes up their experiments with Multi-Cluster Kafka on Kubernetes with Strimzi.

🔥 Apache Flink 2.3.0 has been released, including a Native S3 FileSystem that is detailed in this post.

Francisco Morillo takes a look at the performance of Spark 4.1 Real-time Mode (RTM) compared to Flink.

🔥 Excellent deep-dive from Gorgias' Matthieu Bonneviot about their experience Building Billing on Apache Flink and the impact of event time on the system implementation.

Zander Matheson - Introducing the dbt Adapter for Confluent Cloud Flink SQL.

Katya Gorshkova - Calling LLMs from Flink.

Salva Alcántara has a nice write-up about Multi-Way Joins in Flink’s DataStream API.

Jevin Maltais was a guest on Data Engineering Podcast and talked about many Kafka-related topics including his project TypeStream for declaring Kafka Streams pipelines as config.

Yaroslav Tkachenko has launched Streamling, describing it as "a performant and extensible data streaming runtime built with the RAD stack (Rust, Arrow, DataFusion)".

Gilles Philippart has published a set of Data Streaming Cheatsheets under the banner of StreamSheets, including this one for Flink.

🔥 A deep-dive from Junaid Effendi looking at How Feldera Works: A True Incremental View Maintenance Engine.

On the subject of IVM, Databricks published a paper about Enzyme: Incremental View Maintenance for Data Engineering.

🔥 Kyle Cheung - DuckDB Internals: Why is DuckDB Fast? (Part 1).

🔥 Building ClickCannon - a tool for benchmarking ClickHouse.

It’s ten years since ClickHouse was created, and their blog has a series of interesting posts reflecting on both the tool and the community around it:

🔥 Jack Vanlightly asks: Can We Agree on a Storage/Workload Architecture Taxonomy?

Animesh Kumar et al assert that Most enterprise AI-for-data agents failed in 2025 because they lacked context, and Data Products, built above the engine, are the fix.

A recording and transcript of a talk that OpenAI’s Bonnie Xu did at InfoQ discussing their use of AI Agents to Make Sense of Data at OpenAI.

Pavlina Mitsou and Jonathan Warburton write about The Context Layer Behind Spotify’s Data Assistant.

Chen Chang and colleagues at Anthropic discuss How Anthropic enables self-service data analytics with Claude.

🔥 Fresha continue their run of excellent blog posts, with this one from Daniel Wiszowaty: Everything Everywhere As Of Once: Rebuilding Postgres Inside Snowflake.

Yordan Ivanov - How I Made My Data Platform’s Failures Public and Earned My Stakeholders' Trust.

Goutham Budati - Why Technically Excellent Data Teams Still Fail.

Booking.com’s Tiago Ferreira writes about scaling a shared DWH across multiple teams without turning governance into a delivery bottleneck.

Amer Hesson and colleagues at Netflix discuss Managing Data Assets at Netflix Scale.

Uber’s Daniel Musgrave and colleagues write about Simplifying Data and Product Integrations with a Data Abstraction Layer.

Details of GuideWire’s Federated Query Platform for ML at Scale.

Patrick Lam - Airbnb evolved its data architecture.

Rohit Channe and Simran Mirchandani - How Lyft Governs and Scales Key Data Definitions.

René Luijk discusses the use of the C4 model in the context of data platforms.

🔥 Joe Reis - The Turf Wars Are Over. Time to Cross-Train.

🔥 Ben Rogojan - In 2026 The Data Fundamentals Matter More Than Ever.

Madison Mae - 4 Analytic Engineering Fundamentals That Haven’t Changed.

A reminder of Roche’s Maxim of Data Transformation:

Bruno Masciarelli - How to Build a Simple, Bulletproof Data Pipeline.

Couple of good posts from Joshua Kim about building a pipeline from scratch with dbt part 1 / part 2.

Charly Clairmont - Why dbt-state Matters More Than You Think.

Joachim Hodana - 5 dbt mistakes I see in every startup.

dbtLabs published their 2026 State of Analytics Engineering Report, with some interesting data around use of AI (ofc).

Details of dbt Core v2 from Joel Labes and Grace Goheen.

Vikas Rai describes the Data Profiling Framework at Halodoc.

Artem Golubin - Using local ClickHouse for data processing.

🔥 Gunnar Morling has announced the 1.0 release of Hardwood, which is, in his words, a fast, lightweight Apache Parquet reader for the JVM. He’s published some impressive benchmark figures too, showing just how much of a performance improvement can be had when reading Parquet files by using multi-threading.

vortex-java is a project that builds on Hardwood, implementing the Vortex columnar format in pure Java.

If you want to see first-hand how open-source projects navigate issues that are not straightforward and have strong arguments on either side, this issue on the Iceberg project is fascinating: #14797 Implement Iceberg Kafka Connect with Delta Writer Support in DV Mode.

The Apache Hudi project has been busy, releasing 1.2.0 with support for Stateless Global Upserts for Flink, along with publishing details of how Hudi is used at Southwest Airlines and Penn Entertainment.

Sivabalan Narayanan discusses on the Hudi blog Why Metadata Has to Be Mutation-Friendly.

Iceberg Doctor is a tool from Sarthak Singh for diagnostics on your Iceberg metadata.

A three part series (1, 2, 3) from Giannis Polyzos about Apache Fluss and lakehouse tiering, along with a separate introductory overview to the concepts.

A useful post from Alex Merced about partition evolution in Iceberg.

Two posts from Soumil Shah about shredding VARIANT fields in Iceberg.

🔥 Kelsey Hightower - Thoughts on Open Source (2024).

🔥 Datadog’s Shree Sampath writes about how they implement HA for Postgres on Kubernetes.

Warner Music Group’s Yask Srivastava describes why we shrank our TimescaleDB chunks from 30 days to 7.

Tom Pang - The only scalable delete in Postgres is DROP TABLE.

pg_durable brings durable execution to Postgres.

🔥 This is less "data" blog post and more just "cool tech blog post" :) Conor Gallagher from Zalando details how they do Client-Side Load Balancing at a Million Requests Per Second.

Eric Sun - The Join-Aware Materialized View Query Rewrite Gap.

A good write-up from dbt’s Tristan Handy about recent tech events and trends: Hunting for Tokens. Snowflake Summit. Agent Use Cases.