Consumer offset "stuck" on certain partitions

Observed behavior I consume messages from a topic and this topic has 24 partitions. I started to consume from beginning and at first everything was fine but after some time the consumer stopped consuming messages from certain partitions.

The issue itself is very similar to this issue (562) but I’m using the current version of KafkaJS (v1.15.0) so I’m at a loss what the problem could be. As far as I’m aware the topic also uses log compaction.

I wrote a simple partition assigner that I programmed to only consume from partitions that were “stuck”. After that I added some console.log messages into the KafkaJS code (consumerGroup.js) to debug the problem further. I came to the point that I always got zero messages in the response from broker.fetch.

This was the response:

[
  {
    topicName: "MyTopic",
    partitions: [
      {
        partition: 1,
        errorCode: 0,
        highWatermark: "532672",
        lastStableOffset: "532672",
        lastStartOffset: "0",
        abortedTransactions: [],
        preferredReadReplica: -1,
        messages: [],
      },
    ],
  }
]

The offset that was used to fetch the next messages was like this: { MyTopic: { '1': '484158' } }

There are clearly still messages to consume but it always fetches zero because always the offset 484158 is used. I changed the offset manually via the admin interface to a higher and valid offset and after that the consumer worked again.

Expected behavior I would expect to receive all messages until the latest offset.

Environment:

• OS: Mac OS 10.15.7
• KafkaJS version 1.15.0
• Kafka version 2.6.1
• NodeJS version 12.18.3

Additional context If further logs are needed I can provide them. I couldn’t see any useful debug messages for this problem…