NOT_ENOUGH_REPLICAS Errors After Upgrading The Spring Boot And Spring Cloud Version
See original GitHub issueHello. After I increased the version in my application, I started getting errors. I have 6 active brokers in my Kafka cluster. When I run my application with the old version, I don’t get any errors. However, when I run my application with the new version, I get the following error. I have 6 active brokers, but pretend there aren’t enough brokers to replicate.
I think it’s a bug with version mismatch. Or min.insync.replicas configuration does not work after the new version.
New Project Version : spring-boot-starter-parent : 2.4.13 spring-cloud-dependencies : 2020.0.5
Old Project Version : spring-boot-starter-parent : 2.3.4.RELEASE spring-cloud-dependencies : Hoxton.SR5
Configuration :
spring:
cloud:
stream:
kafka:
bindings:
person-topic-out:
producer:
configuration:
acks: all
retries: 2147483647
enable:
idempotence: true
topic:
properties:
min.insync.replicas: 4
binders:
personKafka:
type: kafka
environment:
spring:
cloud:
stream:
kafka:
binder:
brokers: kafkaAddress:19092
minPartitionCount: 8
autoCreateTopics: true
autoAddPartitions: true
replication-factor: 1
bindings:
person-topic-out:
destination: person-topic
contentType: application/json
binder: personKafka
producer:
partition-count: 5
Error In Spring Boot App :
2022-02-24 21:57:09.677 WARN 12812 --- [ad | producer-1] o.a.k.clients.producer.internals.Sender : [Producer clientId=producer-1] Got error produce response with correlation id 553 on topic-partition person-topic-3, retrying (2147483372 attempts left). Error: NOT_ENOUGH_REPLICAS
2022-02-24 21:57:09.712 WARN 12812 --- [ad | producer-1] o.a.k.clients.producer.internals.Sender : [Producer clientId=producer-1] Got error produce response with correlation id 554 on topic-partition person-topic-6, retrying (2147483375 attempts left). Error: NOT_ENOUGH_REPLICAS
2022-02-24 21:57:09.781 WARN 12812 --- [ad | producer-1] o.a.k.clients.producer.internals.Sender : [Producer clientId=producer-1] Got error produce response with correlation id 555 on topic-partition person-topic-3, retrying (2147483371 attempts left). Error: NOT_ENOUGH_REPLICAS
....
....
2022-02-24 21:57:09.956 ERROR 12812 --- [ad | producer-1] o.s.k.support.LoggingProducerListener : Exception thrown when sending a message with key='null' and payload='byte[46]' to topic person-topic:
org.apache.kafka.common.KafkaException: Producer is closed forcefully.
at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:748) ~[kafka-clients-2.6.3.jar:na]
at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:735) ~[kafka-clients-2.6.3.jar:na]
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:280) ~[kafka-clients-2.6.3.jar:na]
at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]
2022-02-24 21:57:09.956 ERROR 12812 --- [ad | producer-1] o.s.k.support.LoggingProducerListener : Exception thrown when sending a message with key='null' and payload='byte[45]' to topic person-topic:
Error In Kafka Broker :
[2022-02-24 22:10:54,924] ERROR [ReplicaManager broker=6] Error processing append operation on partition person-topic-5 (kafka.server.ReplicaManager)
org.apache.kafka.common.errors.NotEnoughReplicasException: The size of the current ISR Set(6) is insufficient to satisfy the min.isr requirement of 3 for partition person-topic-5
Issue Analytics
- State:
- Created 2 years ago
- Comments:5 (3 by maintainers)
Top Results From Across the Web
Startup error in Spring Cloud Stream after upgrading to Spring ...
I just upgraded a Spring Boot application that uses Spring Cloud Stream Kafka producers and consumers to plugins { id("org.springframework.boot") version ...
Read more >Spring Cloud 2021.0.3 is available. Compatible with Spring ...
Notable Changes in the 2021.0.3 Release Train. This release is primarily for compatibility with Spring Boot 2.7.0 along with Spring Boot 2.6.x.
Read more >Spring Cloud
Generating A New Spring Cloud Project ... The easiest way to get started is visit start.spring.io, select your Spring Boot version and the...
Read more >Spring Cloud 2022.0.0 Release Candidate 3 (codename ...
Notable Changes in the 2022.0.0 RC3 release. Click here to see all issues in this release. Spring Cloud 2022.0.0-RC3 requires Spring Boot 3.0.0....
Read more >Spring Cloud OpenFeign
Spring Cloud integrates Eureka, Spring Cloud CircuitBreaker, as well as Spring Cloud LoadBalancer to provide a load-balanced http client when using Feign.
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
@radoslavCvijetinovic Glad to hear that it worked for you after aligning the configs. With regard to your other question, it is hard to say what happened there. It is very much likely that some combination of broker upgrades + client + app could have caused this. In any case, the
3.0.xversion of Spring Cloud Stream is out of support and you did the right thing by upgrading to3.1.x.I am closing this issue now. Feel free to re-open if there is anything further that needs to be investigated. Thank you!
@omercelikceng @radoslavCvijetinovic We think this is a configuration issue. Something is mismatched either on the broker or at the app level. We definitely made some changes to how
replication-factoris processed between those two Spring Cloud Stream versions (3.0.x/3.1.x). See this PR for more details on that. The error above seems to indicate that the Kafka broker is set with a value of3formin.insync.replicas, but you are overriding that to4on the topic. You also set areplication-factorof1at the binder level. Looks like, something needs to be coordinated here between those properties.