Message consumption acknowledgement in Apache Kafka

Question

I've implemented a Java Consumer that consumes messages from a Kafka topic which are then sent with POST requests to a REST API.

    while (true) {
        ConsumerRecords<String, Object> records = consumer.poll(200);
        for (ConsumerRecord<String, Object> record : records) {
            CloseableHttpClient httpClient = HttpClientBuilder.create().build();  
            Object message = record.value();
            JSONObject jsonObj = new JSONObject(message.toString());
            try {
                HttpPost request = new HttpPost(this.getConsumerProperties().getProperty("api.url"));
                StringEntity params = new StringEntity(message.toString());
                request.addHeader("content-type", "application/json");
                request.addHeader("Accept", "application/json");
                request.setEntity(params);

                CloseableHttpResponse response = httpClient.execute(request);
                HttpEntity entity = response.getEntity();
                String responseString = EntityUtils.toString(entity, "UTF-8");
                System.out.println(message.toString());
                System.out.println(responseString);
            } catch(Exception ex) {
              ex.printStackTrace();
            }  finally {
                try {   
                    httpClient.close(); 
                } catch(IOException ex) {
                    ex.printStackTrace();
                } 
            }                  
        } 
    }   

Say that a message has been consumed, but the Java class failed to reach out the REST API. The message will never be delivered but it will be marked as consumed. What is the best way to handle such cases? Can I somehow acknowledge messages if and only if the response from the REST API was successful?

Answer 1

In the consumer properties, set the enable.auto.commit to false. This would mean that the onus of committing the offset lies with the consumer.

So, in the above example, based on the response.statusCode you may choose to commit the offset by calling consumer.commitAsync().