Kafka Rebalance Loops: max.poll.interval.ms and Slow Consumers

Every consumer instance is up and busy, yet lag keeps growing, the group rebalances every few minutes and downstream sees duplicates. When I see that, I first check whether the poll loop still keeps its contract with the coordinator.

Two liveness checks

A group member is checked in two ways:

• Heartbeats come from a background thread. Silence for session.timeout.ms (45 seconds by default) means the member is dead: a crash or a network problem.
• max.poll.interval.ms (five minutes by default) is the longest allowed gap between two poll() calls. The KafkaConsumer Javadoc calls what it catches a "livelock": the process heartbeats but makes no progress.

Everything between two polls counts against that limit: the whole batch, retries, commits.

while (running) {
    ConsumerRecords<String, OrderEvent> records = consumer.poll(Duration.ofSeconds(1));
    for (ConsumerRecord<String, OrderEvent> record : records) {
        handle(record); // one downstream call per record
    }
    consumer.commitSync(); // reached only after the whole batch
}

A poll() returns up to max.poll.records records (500 by default), and the whole batch must finish inside max.poll.interval.ms: 600 ms per record on average with the defaults. At 20 ms per record a batch takes 10 seconds. If the downstream service slows to one second per call, it takes over eight minutes. Spring's @KafkaListener doesn't change this: its container polls, calls your listener per record, then polls again.

How it turns into a loop

1. The heartbeat thread notices the missed deadline, logs a warning and sends a LeaveGroup request.
2. The group rebalances. Another member takes over the partitions from the last committed offset: the start of the unfinished batch.
3. The evicted consumer, unaware, keeps processing, so for a while two consumers work on the same records.
4. Its commit fails, and its next poll() rejoins the group: another rebalance.
5. The new owner gets the same records, calls the same slow dependency and misses the same deadline.

The evicted consumer logs (prefix trimmed):

WARN  consumer poll timeout has expired. This means the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time processing messages. You can address this either by increasing max.poll.interval.ms or by reducing the maximum size of batches returned in poll() with max.poll.records.
INFO  Member consumer-order-indexer-1-... sending LeaveGroup request to coordinator ... due to consumer poll timeout has expired.

When the batch finishes, commitSync() throws:

org.apache.kafka.clients.consumer.CommitFailedException: Offset commit cannot be completed since the consumer is not part of an active group for auto partition assignment; it is likely that the consumer was kicked out of the group.

The rest of the group logs Request joining group due to: group is already rebalancing, and the generation in Successfully joined group with generation climbs fast. On dashboards, watch the time-between-poll-max metric creep toward the limit, and rebalance-rate-per-hour.

Fixes, in the order I try them

• Make the batch fit. Lower max.poll.records. Fetching is unaffected; the consumer hands out its cached records in smaller chunks.
• Bound the time per record. Put a timeout on every downstream call, and make sure it caps the whole call, which a read timeout doesn't. In-listener retries count against the same budget; long retries belong in a retry topic.
• Raise the limit when slow is legitimate. The cost: a stuck consumer holds its partitions longer, and a rebalance can wait that long for members to reach poll().
• Pause for long work. Hand the batch to a worker thread, pause() the partitions and keep calling poll(), which returns nothing for them but keeps the member in the group. Commit and resume() afterwards. Pause state doesn't survive a rebalance, so your ConsumerRebalanceListener must handle in-flight work.
• Make processing idempotent. Deploys and scaling cause rebalances too, so duplicates arrive anyway (see idempotent consumers).

# worst case per record: 1 s timeout x 2 attempts + backoff, about 2.1 s
# 100 records x 2.1 s = 210 s, inside the 300 s limit
max.poll.records=100
max.poll.interval.ms=300000
partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor

Making unavoidable rebalances cheaper

Cooperative rebalancing. Since Kafka 3.0 the default partition.assignment.strategy is [RangeAssignor, CooperativeStickyAssignor], which still runs the eager range assignor: every rebalance makes every member give up all its partitions. From that default, one rolling bounce that removes RangeAssignor switches the group to cooperative-sticky, where only the partitions that move are revoked.

Static membership. With a stable group.instance.id, a member that restarts within session.timeout.ms (raise it to cover a restart) gets its partitions back without a rebalance. The ID must survive the restart: a StatefulSet pod name works, a random Deployment pod name doesn't. The flip side: static members don't send LeaveGroup, so the partitions of a dead member, or of one that missed max.poll.interval.ms, move only after session.timeout.ms.

What about KIP-848?

Kafka 4.0 (March 2025) made the KIP-848 consumer group protocol generally available: on by default on the broker, opt-in on the client with group.protocol=consumer. Rebalances become incremental and broker-driven, and session timeout, heartbeat interval and assignor move to the broker. max.poll.interval.ms stays on the client, and the new consumer logs almost the same warning before leaving the group, so the arithmetic doesn't change.

Takeaways

• max.poll.records × worst-case time per record must fit in max.poll.interval.ms with headroom.
• An evicted consumer keeps processing until it commits or polls. Plan for duplicates.
• Alert on time-between-poll-max and the rebalance rate, not only on lag.
• Cooperative-sticky and static membership make rebalances cheaper, not slow loops faster.