Randomly get topic ... whcih is not assigned to any partitions exception

See original GitHub issue

Bug Report

Current behavior

I am using NestJs with Kafka and the error The client consumer subscribed to the topic (${topic}) whcih is not assigned to any partitions., is often thrown with failed code execution Some other time I get The client consumer did not subscribe to the corresponding reply topic (${topic}).,

The setup is very basic

Input Code

  @Client(serviceConfig)
  streamClient;

  onModuleInit() {
      this.streamClient.subscribeToResponseOf(...);
    };
  }

  onModuleDestroy() {
    this.streamClient.close();
  }


export const serviceConfig = {
  transport: Transport.KAFKA,
  options: {
    client: {
      clientId: '...',
      brokers: [`...`],
    },
    consumer: {
      groupId: '...',
    },
  },
};

Expected behavior

The code often runs fine but sometimes the error just happens, specifically around the time of rebalancing

My configuration consists of 3 kafka broker and 2 NestJS instances running in paralell with same source code / same setting (kubernete setup)

I put also my situation here

https://stackoverflow.com/questions/65788607/is-it-normal-to-have-less-partition-than-broker-count-kafka-configuration?noredirect=1#comment116317902_65788607

Possible Solution

I see that my NestJS version is before this commit https://github.com/nestjs/nest/commit/b0a0623cf8374c2dca19a5aaff1ded19ebda5926#diff-72a9f557d4f070b2a084eaf4ba5d7e3ad02a496f670958e6efee128624e6b097

I am not sure if the behaviour i am facing is related to what the commit is fixxing?

Environment


    "@nestjs/common": "^7.0.0",
    "@nestjs/core": "^7.0.0",
    "@nestjs/microservices": "^7.4.2",

Thanks for the help!

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Comments:6 (2 by maintainers)

github_iconTop GitHub Comments

6reactions
smilovanoviccommented, Mar 24, 2022

Why was this closed? Having a unique consumer group per instance is not really a solution because while you want all instances to consume the reply message you do not want that behaviour for regular messages which should be processed only once.

1reaction
qkhanhprocommented, Jan 19, 2021

I am afraid this happens with specific HA configuration so it would be kind of hard to provide a minimal repo. I will still try to create a minimal reproduction and update when possible Thanks a lot @Tony133

Read more comments on GitHub >

github_iconTop Results From Across the Web

IllegalStateException Subscription to topics, partitions and ...
Are you able to get the right mapping of topic and partition from consumer.assignment() ? Also, the functions are void methods. Is this...
Read more >
Understanding Kafka partition assignment strategies and how ...
In a previous blog post, I explain how the Apache Kafka Rebalance Protocol does work and how is internally used.
Read more >
Chapter 4. Kafka Consumers: Reading Data from Kafka
This happens whenever Range assignment is used and the number of consumers does not divide the number of partitions in each topic neatly....
Read more >
KafkaConsumer (kafka 2.4.0 API)
To get semantics similar to a queue in a traditional messaging system all ... Manual partition assignment does not use group coordination, so...
Read more >
5 Common Pitfalls When Using Apache Kafka - Confluent
Thus, if you have 10 topics each with 50 partitions, there are at least 2,000 file handlers at any given time. To put...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found