rmoff's random ramblings
about talks

Reading Kafka Connect Offsets via the REST Proxy

Published May 2, 2019 by in Kafka Connect, Kafka REST Proxy at https://rmoff.net/2019/05/02/reading-kafka-connect-offsets-via-the-rest-proxy/

When you run Kafka Connect in distributed mode it uses a Kafka topic to store the offset information for each connector. Because it’s just a Kafka topic, you can read that information using any consumer.

It may be that you want to query this information. You can use a CLI tool like kafkacat:

$ kafkacat -b localhost:9092 -t docker-kafka-connect-offsets -C -K:
["source-debezium-orders-00",{"server":"asgard"}]:{"file":"mysql-bin.000003","pos":154}
["source-debezium-orders-00",{"server":"asgard"}]:{"ts_sec":1556791017,"file":"mysql-bin.000003","pos":9041,"row":1,"server_id":223344,"event":2}

You may want to access this information over HTTP though. Here’s how, using the Confluent REST Proxy. For more cheat-sheet of the REST proxy see here.

  • Create a consumer

    echo '{"name": "rmoff_consumer_instance", "format": "json", "auto.offset.reset": "earliest"}' | \
      http POST http://localhost:8082/consumers/rmoff_consumer_group \
      Content-Type:application/vnd.kafka.v2+json
  • Create a subscription to the offset topic for the consumer. Make sure you put the correct topic name here

    echo '{"topics":["docker-kafka-connect-offsets"]}' | \
      http POST http://localhost:8082/consumers/rmoff_consumer_group/instances/rmoff_consumer_instance/subscription \
      Content-Type:application/vnd.kafka.v2+json
  • Read any new messages from the topic

    http http://localhost:8082/consumers/rmoff_consumer_group/instances/rmoff_consumer_instance/records \
      Accept:application/vnd.kafka.json.v2+json
    HTTP/1.1 200 OK
    Content-Encoding: gzip
    Content-Length: 154
    Content-Type: application/vnd.kafka.json.v2+json
    Date: Thu, 02 May 2019 09:50:27 GMT
    Server: Jetty(9.4.14.v20181114)
    Vary: Accept-Encoding, User-Agent
    
    [
        {
            "key": [
                "source-debezium-orders-00",
                {
                    "server": "asgard"
                }
            ],
            "offset": 0,
            "partition": 20,
            "topic": "docker-kafka-connect-offsets",
            "value": {
                "file": "mysql-bin.000003",
                "pos": 154
            }
        }
    ]

Robin Moffatt

Robin Moffatt works on the DevRel team at Confluent. He likes writing about himself in the third person, eating good breakfasts, and drinking good beer.

Story logo

© 2025