如何在融合的 kafka 休息代理中访问主题的最新偏移量以计算滞后

Question

在融合的 kafka 休息代理中，我们可以获得特定消费者组的最后提交的偏移量，但是我们如何获得主题的最新偏移量来计算滞后。

Answer 1

You can use Kafka REST Proxy to fetch the latest offset committed for a particular partition . According to the Confluent Docs ,

 GET /consumers/(string: group_name)/instances/(string: instance)/offsets

Get the last committed offsets for the given partitions (whether the commit happened by this process or another).

Note that this request must be made to the specific REST proxy instance holding the consumer instance.

Parameters:

• group_name (string) -- The name of the consumer group

• instance (string) -- The ID of the consumer instance Request JSON

Array of Objects:

• partitions -- A list of partitions to find the last committed offsets for
• partitions[i].topic (string) -- Name of the topic
• partitions[i].partition (int) -- Partition ID

Response JSON Array of Objects:

• offsets -- A list of committed offsets
• offsets[i].topic (string) -- Name of the topic for which an offset was committed
• offsets[i].partition (int) -- Partition ID for which an offset was committed
• offsets[i].offset (int) -- Committed offset
• offsets[i].metadata (string) -- Metadata for the committed offset

Status Codes:

• 404 Not Found --
• Error code 40402 -- Partition not found
• Error code 40403 -- Consumer instance not found

Example Request:

GET /consumers/testgroup/instances/my_consumer/offsets HTTP/1.1
Host: proxy-instance.kafkaproxy.example.com
Accept: application/vnd.kafka.v2+json, application/vnd.kafka+json, application/json

{
  "partitions": [
    {
      "topic": "test",
      "partition": 0
    },
    {
      "topic": "test",
      "partition": 1
    }

  ]
}

Example Response:

HTTP/1.1 200 OK
Content-Type: application/vnd.kafka.v2+json

{"offsets":
 [
  {
    "topic": "test",
    "partition": 0,
    "offset": 21,
    "metadata":""
  },
  {
    "topic": "test",
    "partition": 1,
    "offset": 31,
    "metadata":""
  }
 ]
}