Today, we ran into an unexpected issue while working with Kafka. One of the topics we were consuming suddenly published over 50,000 messages, and we encountered the following error on our end:
Error:
[Consumer clientId=xyz_1] Asynchronous auto-commit of offsets failed : Offset commit cannot be completed since the consumer is not part of an active group of auto partition assignment; it is likely that the consumer was kicked out of the group.. will continue to join the group
Root cause
We first verified whether any recent changes had been made on the consumer side and confirmed that there were no updates on either the consumer or producer.
Next, we investigated what might have triggered the issue. By reviewing our Grafana monitoring for the topic, we noticed that on the day the error occurred, over 50,000 messages were published within a 2-hour window.
From this, we inferred that the message processing time had increased significantly. As a result, by the time the consumer attempted to commit the offset, the consumer group had already been disconnected, causing the same messages to be reprocessed repeatedly.
To fix this issue
- We checked with Producer for specific attribute which can narrow down the messages for the consumer. (Additional filter)
- We have updated Spring Kafka properties as below :
Spring:
Kafka:
max:
poll:
records : 100 (default was 500)
interval:
ms: 600000 (default was 300000 = 5m)What these property does?
max.poll.interval.ms
A simple way to address this issue is to increase the max.poll.interval.ms value, allowing the consumer more time to process records.
max.poll.records
The maximum number of records a poll will return
Conclusion
This is a common pitfall when working with Apache Kafka. It’s important to configure this parameter thoughtfully — setting it too high can lead to increased latency and reduced throughput.