ERROR: The group is rebalancing, so a rejoin is needed

See original GitHub issue

Hi,

i need some help with the error message from the title. I read this issue, which seems to be related:

https://github.com/tulios/kafkajs/issues/226

I also read the documentation here, which also seems to be related: https://kafka.js.org/docs/faq#what-does-it-mean-to-get-rebalance_in_progress-errors

However, it doesnt matter what i try the errors are remaining. It only occures when i redeploy my application and unfortunately it always triggers our alerting system, which is annoying.

Here is the log from my application:

{"loglevel":"error","msg":"ERROR [Connection] Response Heartbeat(key: 12, version: 1) {\"timestamp\":\"2020-04-06T10:32:41.900Z\",\"logger\":\"kafkajs\",\"broker\":\"kafka-2:9093\",\"clientId\":\"my-custom-client-id-server\",\"error\":\"The group is rebalancing, so a rejoin is needed\",\"correlationId\":16,\"size\":10}"}
{"loglevel":"error","msg":"ERROR [Runner] The group is rebalancing, re-joining {\"timestamp\":\"2020-04-06T10:32:41.901Z\",\"logger\":\"kafkajs\",\"groupId\":\"my-custom-client-id-v001-server\",\"memberId\":\"my-custom-client-id-server-bea7349c-32f2-4855-92a2-c2549f172082\",\"error\":\"The group is rebalancing, so a rejoin is needed\",\"retryCount\":0,\"retryTime\":3560}"}
{"loglevel":"error","msg":"ERROR [Connection] Response SyncGroup(key: 14, version: 1) {\"timestamp\":\"2020-04-06T10:33:41.906Z\",\"logger\":\"kafkajs\",\"broker\":\"kafka-2:9093\",\"clientId\":\"my-custom-client-id-server\",\"error\":\"The group is rebalancing, so a rejoin is needed\",\"correlationId\":18,\"size\":14}"}
{"loglevel":"info","msg":"INFO [Runner] Consumer has joined the group {\"timestamp\":\"2020-04-06T10:35:41.919Z\",\"logger\":\"kafkajs\",\"groupId\":\"my-custom-client-id-v001-server\",\"memberId\":\"my-custom-client-id-server-bea7349c-32f2-4855-92a2-c2549f172082\",\"leaderId\":\"my-custom-client-id-server-0b9827ef-9e82-4343-96d7-3bee2053c449\",\"isLeader\":false,\"memberAssignment\":{\"my-topic-1\":[11,15,19,23,27,3,31,35,39,7],\"my-topic-2\":[11,15,19,23,27,3,31,35,39,7]},\"groupProtocol\":\"RoundRobinAssigner\",\"duration\":180018}"}
{"loglevel":"error","msg":"ERROR [Connection] Response Heartbeat(key: 12, version: 1) {\"timestamp\":\"2020-04-06T10:37:17.024Z\",\"logger\":\"kafkajs\",\"broker\":\"kafka-2:9093\",\"clientId\":\"my-custom-client-id-server\",\"error\":\"The group is rebalancing, so a rejoin is needed\",\"correlationId\":45,\"size\":10}"}
{"loglevel":"error","msg":"ERROR [Runner] The group is rebalancing, re-joining {\"timestamp\":\"2020-04-06T10:37:17.025Z\",\"logger\":\"kafkajs\",\"groupId\":\"my-custom-client-id-v001-server\",\"memberId\":\"my-custom-client-id-server-bea7349c-32f2-4855-92a2-c2549f172082\",\"error\":\"The group is rebalancing, so a rejoin is needed\",\"retryCount\":0,\"retryTime\":2618}"}
{"loglevel":"info","msg":"INFO [Runner] Consumer has joined the group {\"timestamp\":\"2020-04-06T10:37:41.916Z\",\"logger\":\"kafkajs\",\"groupId\":\"my-custom-client-id-v001-server\",\"memberId\":\"my-custom-client-id-server-bea7349c-32f2-4855-92a2-c2549f172082\",\"leaderId\":\"my-custom-client-id-server-69d3bcd8-a1d9-428f-8742-142ccb674b67\",\"isLeader\":false,\"memberAssignment\":{\"my-topic-1\":[1,11,13,15,17,19,21,23,25,27,29,3,31,33,35,37,39,5,7,9],\"my-topic-2\":[1,11,13,15,17,19,21,23,25,27,29,3,31,33,35,37,39,5,7,9]},\"groupProtocol\":\"RoundRobinAssigner\",\"duration\":24891}"}

I already tried to increase the following config parameters.

maxRetryTime,sessionTimeout, rebalanceTimeout,heartbeatInterval

Neither helped, any hints would be greatly apprechiated.

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Reactions:12
  • Comments:8

github_iconTop GitHub Comments

62reactions
cristianbriscarucommented, Nov 27, 2020

I too believe that rebalancing should not be logged as error given that it is part of the normal protocol/workflow for kafka when a consumer joins/leaves a consumer group. No other libraries for kafka log this as an error ( springboot, kafka-node,…) . I believe this should be at worst logged as a warning if not a info.

30reactions
ayozemrcommented, Sep 16, 2021

The thing is, they are errors, which is why they are logged as such. It’s just that sometimes they are expected errors. However, only you have the context to determine when it is fine (during deploys) and when it isn’t. I would suggest that you adjust your alerting to not alert during deploys.

At infrastructure/devops world, an error means an alert that could ring someone’s phone at night… So if its a “normal error” that is not urgent, means it should be a warning, not an error.

+1 on changing that from ERR to WARN

Thanks!

Read more comments on GitHub >

github_iconTop Results From Across the Web

Kafkajs - `The group is rebalancing, so a rejoin is needed ...
I increased sessionTimeout & heartbeatInteval to higher values and different combinations, but still under heavy message load, I get the error.
Read more >
Solving My Weird Kafka Rebalancing Problems & Explaining ...
In our example, the consumer has a downtime but can rejoin the group inside the configured time limit's bounds. Thus, no rebalancing is...
Read more >
Understanding Kafka's Consumer Group Rebalancing - Verica
1. Consumption fully stops while the consumer group rebalances the partitions. · 2. If the consumer failure is transient and it rejoins the...
Read more >
FAQ - KafkaJS
The rebalancing state is enforced on the broker side. When a consumer tries to commit offsets, the broker will respond with REBALANCE_IN_PROGRESS ....
Read more >
Azure Event hub Kafka rebalancing issue - Microsoft Q&A
Join group failed with org.apache.kafka.common.errors.RebalanceInProgressException: The group is rebalancing, so a rejoin is needed.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found