ERROR: The group is rebalancing, so a rejoin is needed
See original GitHub issueHi,
i need some help with the error message from the title. I read this issue, which seems to be related:
https://github.com/tulios/kafkajs/issues/226
I also read the documentation here, which also seems to be related: https://kafka.js.org/docs/faq#what-does-it-mean-to-get-rebalance_in_progress-errors
However, it doesnt matter what i try the errors are remaining. It only occures when i redeploy my application and unfortunately it always triggers our alerting system, which is annoying.
Here is the log from my application:
{"loglevel":"error","msg":"ERROR [Connection] Response Heartbeat(key: 12, version: 1) {\"timestamp\":\"2020-04-06T10:32:41.900Z\",\"logger\":\"kafkajs\",\"broker\":\"kafka-2:9093\",\"clientId\":\"my-custom-client-id-server\",\"error\":\"The group is rebalancing, so a rejoin is needed\",\"correlationId\":16,\"size\":10}"}
{"loglevel":"error","msg":"ERROR [Runner] The group is rebalancing, re-joining {\"timestamp\":\"2020-04-06T10:32:41.901Z\",\"logger\":\"kafkajs\",\"groupId\":\"my-custom-client-id-v001-server\",\"memberId\":\"my-custom-client-id-server-bea7349c-32f2-4855-92a2-c2549f172082\",\"error\":\"The group is rebalancing, so a rejoin is needed\",\"retryCount\":0,\"retryTime\":3560}"}
{"loglevel":"error","msg":"ERROR [Connection] Response SyncGroup(key: 14, version: 1) {\"timestamp\":\"2020-04-06T10:33:41.906Z\",\"logger\":\"kafkajs\",\"broker\":\"kafka-2:9093\",\"clientId\":\"my-custom-client-id-server\",\"error\":\"The group is rebalancing, so a rejoin is needed\",\"correlationId\":18,\"size\":14}"}
{"loglevel":"info","msg":"INFO [Runner] Consumer has joined the group {\"timestamp\":\"2020-04-06T10:35:41.919Z\",\"logger\":\"kafkajs\",\"groupId\":\"my-custom-client-id-v001-server\",\"memberId\":\"my-custom-client-id-server-bea7349c-32f2-4855-92a2-c2549f172082\",\"leaderId\":\"my-custom-client-id-server-0b9827ef-9e82-4343-96d7-3bee2053c449\",\"isLeader\":false,\"memberAssignment\":{\"my-topic-1\":[11,15,19,23,27,3,31,35,39,7],\"my-topic-2\":[11,15,19,23,27,3,31,35,39,7]},\"groupProtocol\":\"RoundRobinAssigner\",\"duration\":180018}"}
{"loglevel":"error","msg":"ERROR [Connection] Response Heartbeat(key: 12, version: 1) {\"timestamp\":\"2020-04-06T10:37:17.024Z\",\"logger\":\"kafkajs\",\"broker\":\"kafka-2:9093\",\"clientId\":\"my-custom-client-id-server\",\"error\":\"The group is rebalancing, so a rejoin is needed\",\"correlationId\":45,\"size\":10}"}
{"loglevel":"error","msg":"ERROR [Runner] The group is rebalancing, re-joining {\"timestamp\":\"2020-04-06T10:37:17.025Z\",\"logger\":\"kafkajs\",\"groupId\":\"my-custom-client-id-v001-server\",\"memberId\":\"my-custom-client-id-server-bea7349c-32f2-4855-92a2-c2549f172082\",\"error\":\"The group is rebalancing, so a rejoin is needed\",\"retryCount\":0,\"retryTime\":2618}"}
{"loglevel":"info","msg":"INFO [Runner] Consumer has joined the group {\"timestamp\":\"2020-04-06T10:37:41.916Z\",\"logger\":\"kafkajs\",\"groupId\":\"my-custom-client-id-v001-server\",\"memberId\":\"my-custom-client-id-server-bea7349c-32f2-4855-92a2-c2549f172082\",\"leaderId\":\"my-custom-client-id-server-69d3bcd8-a1d9-428f-8742-142ccb674b67\",\"isLeader\":false,\"memberAssignment\":{\"my-topic-1\":[1,11,13,15,17,19,21,23,25,27,29,3,31,33,35,37,39,5,7,9],\"my-topic-2\":[1,11,13,15,17,19,21,23,25,27,29,3,31,33,35,37,39,5,7,9]},\"groupProtocol\":\"RoundRobinAssigner\",\"duration\":24891}"}
I already tried to increase the following config parameters.
maxRetryTime,sessionTimeout, rebalanceTimeout,heartbeatInterval
Neither helped, any hints would be greatly apprechiated.
Issue Analytics
- State:
- Created 3 years ago
- Reactions:12
- Comments:8
Top Results From Across the Web
Kafkajs - `The group is rebalancing, so a rejoin is needed ...
I increased sessionTimeout & heartbeatInteval to higher values and different combinations, but still under heavy message load, I get the error.
Read more >Solving My Weird Kafka Rebalancing Problems & Explaining ...
In our example, the consumer has a downtime but can rejoin the group inside the configured time limit's bounds. Thus, no rebalancing is...
Read more >Understanding Kafka's Consumer Group Rebalancing - Verica
1. Consumption fully stops while the consumer group rebalances the partitions. · 2. If the consumer failure is transient and it rejoins the...
Read more >FAQ - KafkaJS
The rebalancing state is enforced on the broker side. When a consumer tries to commit offsets, the broker will respond with REBALANCE_IN_PROGRESS ....
Read more >Azure Event hub Kafka rebalancing issue - Microsoft Q&A
Join group failed with org.apache.kafka.common.errors.RebalanceInProgressException: The group is rebalancing, so a rejoin is needed.
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
I too believe that rebalancing should not be logged as error given that it is part of the normal protocol/workflow for kafka when a consumer joins/leaves a consumer group. No other libraries for kafka log this as an error ( springboot, kafka-node,…) . I believe this should be at worst logged as a warning if not a info.
At infrastructure/devops world, an error means an alert that could ring someone’s phone at night… So if its a “normal error” that is not urgent, means it should be a warning, not an error.
+1 on changing that from ERR to WARN
Thanks!