"Specified group generation id is not valid" after broker maintenance, consumer stops receiving events

See original GitHub issue

Hi, we are having an issue similar to https://github.com/tulios/kafkajs/issues/1009 but it happens after a broker maintenance. We have consumers running parallelly on different machines, with a heartbeat check triggered on eachBatch. We consume multiple topics, with a specific instance of our service per topic. All of this works fine but we had issues (twice already) when brokers go on maintenance. Some of the instance (thus some of the topics) stop consuming events, but don’t throw errors nor crash (if it crashed we would respawn and everything would be ok). We do see the error message: [Consumer] Crash: KafkaJSNonRetriableError: Specified group generation id is not valid But it doesn’t actually crash, and the instance is stale, it won’t consume any new message or trigger the heartbeat. If we restart the instance it will consume all pending traffic (given the offset is still current). Odd thing is some of the topics keep working fine after the maintenance, so the overall system seems to be “up” unless we check each specific topic.

Issue Analytics

  • State:open
  • Created a year ago
  • Reactions:4
  • Comments:10

github_iconTop GitHub Comments

2reactions
jakewinscommented, Nov 2, 2022

Ran into this as well, proposed fix: https://github.com/tulios/kafkajs/pull/1474

2reactions
IvanRogovskiycommented, Oct 20, 2022

I have pretty the same thing. I have a connection to 11 topics and when I start receiving messages i see the logs below

{"level":"WARN","timestamp":"2022-10-05T08:27:56.258Z","logger":"kafkajs","message":"[ConsumerGroup] Topic has been updated, resync group"


{"level":"ERROR","timestamp":"2022-10-05T08:27:58.856Z","logger":"kafkajs","message":"[Connection] Response SyncGroup(key: 14, version: 3)", error":"Specified group generation id is not valid","correlationId":87,"size":14}

and after it the message that the consumer has been stopped. Increasing of heartbeats interval and sessionTimeout didn’t help

Read more comments on GitHub >

github_iconTop Results From Across the Web

Broker: Specified group generation id is not valid
Problem. First, one of my consumers encounters the following exception. Confluent.Kafka.KafkaException: Broker: Specified group generation id is ...
Read more >
kafka - The Go Programming Language
Package kafka provides high-level Apache Kafka producer and consumers using ... ErrIllegalGeneration Broker: Specified group generation id is not valid ...
Read more >
Troubleshooting your Amazon MSK cluster
Consumer group stuck in PreparingRebalance state; Error delivering broker logs to Amazon CloudWatch Logs; No default security group; Cluster appears stuck ...
Read more >
Kafka protocol guide - Apache Kafka
Produce request specified an invalid value for required acks. ILLEGAL_GENERATION, 22, False, Specified group generation id is not valid.
Read more >
Database Engine events and errors - SQL Server
ls is not a valid broker name. 1094, 15, No, Cannot specify a schema name as a prefix to the trigger name for...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found