NOT_ENOUGH_REPLICAS Errors After Upgrading The Spring Boot And Spring Cloud Version

See original GitHub issue

Hello. After I increased the version in my application, I started getting errors. I have 6 active brokers in my Kafka cluster. When I run my application with the old version, I don’t get any errors. However, when I run my application with the new version, I get the following error. I have 6 active brokers, but pretend there aren’t enough brokers to replicate.

I think it’s a bug with version mismatch. Or min.insync.replicas configuration does not work after the new version.

New Project Version : spring-boot-starter-parent : 2.4.13 spring-cloud-dependencies : 2020.0.5

Old Project Version : spring-boot-starter-parent : 2.3.4.RELEASE spring-cloud-dependencies : Hoxton.SR5

Configuration :

spring:
  cloud:
    stream:
      kafka:
        bindings:
          person-topic-out:
            producer:
              configuration:
                acks: all
                retries: 2147483647
                enable:
                  idempotence: true
              topic:
                properties:
                  min.insync.replicas: 4
      binders:
        personKafka:
          type: kafka
          environment:
            spring:
              cloud:
                stream:
                  kafka:
                    binder:
                      brokers: kafkaAddress:19092
                      minPartitionCount: 8
                      autoCreateTopics: true
                      autoAddPartitions: true
                      replication-factor: 1
      bindings:
        person-topic-out:
          destination: person-topic
          contentType: application/json
          binder: personKafka
          producer:
            partition-count: 5

Error In Spring Boot App :

2022-02-24 21:57:09.677  WARN 12812 --- [ad | producer-1] o.a.k.clients.producer.internals.Sender  : [Producer clientId=producer-1] Got error produce response with correlation id 553 on topic-partition person-topic-3, retrying (2147483372 attempts left). Error: NOT_ENOUGH_REPLICAS
2022-02-24 21:57:09.712  WARN 12812 --- [ad | producer-1] o.a.k.clients.producer.internals.Sender  : [Producer clientId=producer-1] Got error produce response with correlation id 554 on topic-partition person-topic-6, retrying (2147483375 attempts left). Error: NOT_ENOUGH_REPLICAS
2022-02-24 21:57:09.781  WARN 12812 --- [ad | producer-1] o.a.k.clients.producer.internals.Sender  : [Producer clientId=producer-1] Got error produce response with correlation id 555 on topic-partition person-topic-3, retrying (2147483371 attempts left). Error: NOT_ENOUGH_REPLICAS
....
....
2022-02-24 21:57:09.956 ERROR 12812 --- [ad | producer-1] o.s.k.support.LoggingProducerListener    : Exception thrown when sending a message with key='null' and payload='byte[46]' to topic person-topic:

org.apache.kafka.common.KafkaException: Producer is closed forcefully.
	at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:748) ~[kafka-clients-2.6.3.jar:na]
	at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:735) ~[kafka-clients-2.6.3.jar:na]
	at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:280) ~[kafka-clients-2.6.3.jar:na]
	at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]

2022-02-24 21:57:09.956 ERROR 12812 --- [ad | producer-1] o.s.k.support.LoggingProducerListener    : Exception thrown when sending a message with key='null' and payload='byte[45]' to topic person-topic:

Error In Kafka Broker :

[2022-02-24 22:10:54,924] ERROR [ReplicaManager broker=6] Error processing append operation on partition person-topic-5 (kafka.server.ReplicaManager)
org.apache.kafka.common.errors.NotEnoughReplicasException: The size of the current ISR Set(6) is insufficient to satisfy the min.isr requirement of 3 for partition person-topic-5

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:5 (3 by maintainers)

github_iconTop GitHub Comments

1reaction
sobychackocommented, Mar 17, 2022

@radoslavCvijetinovic Glad to hear that it worked for you after aligning the configs. With regard to your other question, it is hard to say what happened there. It is very much likely that some combination of broker upgrades + client + app could have caused this. In any case, the 3.0.x version of Spring Cloud Stream is out of support and you did the right thing by upgrading to 3.1.x.

I am closing this issue now. Feel free to re-open if there is anything further that needs to be investigated. Thank you!

1reaction
sobychackocommented, Mar 16, 2022

@omercelikceng @radoslavCvijetinovic We think this is a configuration issue. Something is mismatched either on the broker or at the app level. We definitely made some changes to how replication-factor is processed between those two Spring Cloud Stream versions (3.0.x/3.1.x). See this PR for more details on that. The error above seems to indicate that the Kafka broker is set with a value of 3 for min.insync.replicas, but you are overriding that to 4 on the topic. You also set a replication-factor of 1 at the binder level. Looks like, something needs to be coordinated here between those properties.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Startup error in Spring Cloud Stream after upgrading to Spring ...
I just upgraded a Spring Boot application that uses Spring Cloud Stream Kafka producers and consumers to plugins { id("org.springframework.boot") version ...
Read more >
Spring Cloud 2021.0.3 is available. Compatible with Spring ...
Notable Changes in the 2021.0.3 Release Train. This release is primarily for compatibility with Spring Boot 2.7.0 along with Spring Boot 2.6.x.
Read more >
Spring Cloud
Generating A New Spring Cloud Project ... The easiest way to get started is visit start.spring.io, select your Spring Boot version and the...
Read more >
Spring Cloud 2022.0.0 Release Candidate 3 (codename ...
Notable Changes in the 2022.0.0 RC3 release. Click here to see all issues in this release. Spring Cloud 2022.0.0-RC3 requires Spring Boot 3.0.0....
Read more >
Spring Cloud OpenFeign
Spring Cloud integrates Eureka, Spring Cloud CircuitBreaker, as well as Spring Cloud LoadBalancer to provide a load-balanced http client when using Feign.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found