There is an open issue for support of EXPLODE/UNNEST functionality in KSQL, and if you need it then do up-vote the issue. Here I detail a hacky, but effective, workaround for exploding arrays into multiple messages—so long as you know the upper-bound on your array.
Kafka Connect is a API within Apache Kafka and its modular nature makes it powerful and flexible. Converters are part of the API but not always fully understood. I’ve written previously about Kafka Connect converters, and this post is just a hands-on example to show even further what they are—and are not—about.
|
Note
|
To understand more about Kafka Connect in general, check out my talk from Kafka Summit London From Zero to Hero with Kafka Connect. |
When you run Kafka Connect in distributed mode it uses a Kafka topic to store the offset information for each connector. Because it’s just a Kafka topic, you can read that information using any consumer.
Prompted by a question on StackOverflow, the requirement is to take a series of events related to a common key and for each key output a series of aggregates derived from a changing value in the events. I’ll use the data from the question, based on ticket statuses. Each ticket can go through various stages, and the requirement was to show, per customer, how many tickets are currently at each stage.
Confluent Cloud now includes a secured Schema Registry, which you can use from external applications, including KSQL.
To configure KSQL for it you need to set:
ksql.schema.registry.url=https://<Schema Registry endpoint>
ksql.schema.registry.basic.auth.credentials.source=USER_INFO
ksql.schema.registry.basic.auth.user.info=<Schema Registry API Key>:<Schema Registry API Secret>What can you use stream-stream joins for? Can you use them to join between a stream of orders and stream of related shipments to do useful things? What’s not supported in KSQL, where are the cracks?
Before you can drop a stream or table that’s populated by a query in KSQL, you have to terminate any queries upon which the object is dependent. Here’s a bit of jq & xargs magic to terminate all queries that are currently running
This post is the companion to an earlier one that I wrote about conference abstracts. In the same way that the last one was inspired by reviewing a ton of abstracts and noticing a recurring pattern in my suggestions, so this one comes from reviewing a bunch of slide decks for a forthcoming conference. They all look like good talks, but in several cases these great talks are fighting to get out from underneath the deadening weight of slides.
Herewith follows my highly-opinionated, fairly-subjective, and extremely-terse advice and general suggestions for slide decks. You can also find relating ramblings in this recent post too. My friend and colleague Vik Gamov also wrote a good post on this same topic, and linked to a good video that I’d recommend you watch.
This shows how to use httpie with the Confluent REST Proxy.
echo '{"records":[{"value":{"foo":"bar"}}]}' | \
http POST http://localhost:8082/topics/jsontest \
Content-Type:application/vnd.kafka.json.v2+json Accept:application/vnd.kafka.v2+jsonA script I’d batch-run on my Markdown files had inserted a UTF-8 non-breaking-space between Markdown heading indicator and the text, which meant that # My title actually got rendered as that, instead of an H3 title.
Looking at the file contents, I could see it wasn’t just a space between the # and the text, but a non-breaking space.
KSQL is generally case-sensitive. Very sensitive, at times ;-)
Full reference is here
The Schema Registry support a REST API for finding out information about the schemas within it. Here’s a quick cheatsheat with REST calls that I often use.
I use Docker and Docker Compose a lot. Like, every day. It’s a fantastic way to build repeatable demos and examples, that can be torn down and spun up in a repeatable way. But…what happens when the demo that was working is spun up and then tail spins down in a blaze of flames?
Tiny little snippet this one. Given a list of images:
$ docker images|grep confluent
confluentinc/cp-enterprise-kafka 5.0.0 d0c5528d7f99 3 months ago 600MB
confluentinc/cp-kafka 5.0.0 373a4e31e02e 3 months ago 558MB
confluentinc/cp-zookeeper 5.0.0 3cab14034c43 3 months ago 558MB
confluentinc/cp-ksql-server 5.0.0 691bc3c1991f 4 months ago 493MB
confluentinc/cp-ksql-cli 5.0.0 e521f3e787d6 4 months ago 488MB
…Now there’s a new version available, and you want to pull down all the latest ones for it:
docker images|grep "^confluentinc"|awk '{print $1}'|xargs -Ifoo docker pull foo:5.1.0