WebClient fails with Pool#acquire(Duration) has been pending for more than the configured timeout of 45000ms after 33000 request in loop
See original GitHub issueI’ve tried write a simple load test with WebClient. One thread with simple loop, 100k iterations, but after 2000 or 3000 cycles I receive a message:
Apr 09, 2020 6:00:38 AM io.netty.util.ResourceLeakDetector reportTracedLeak
SEVERE: LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records:
Created at:
io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:363)
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:139)
io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:147)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:748)
After all 100k iterations, I’ve received only 33k successful responses, and no one was failed. My code written to wait for all responses and prints success/failed requests after one second of sleep. After 45 seconds, I’ve received other 67k failed requests with exception:
reactor.netty.internal.shaded.reactor.pool.PoolAcquireTimeoutException: Pool#acquire(Duration) has been pending for more than the configured timeout of 45000ms
at reactor.netty.internal.shaded.reactor.pool.AbstractPool$Borrower.run(AbstractPool.java:334)
Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
|_ checkpoint ⇢ Request to GET http://localhost:8080/users/50725/5362 [DefaultWebClient]
Stack trace:
at reactor.netty.internal.shaded.reactor.pool.AbstractPool$Borrower.run(AbstractPool.java:334)
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
And my stats looks like
test results, success: 33,085, errors: 66,914
Note that 33085 + 66914 = 99999
Steps to Reproduce
I wrote a simple app with spring-webflux:5.2.5 and reactor-netty:0.9.6:
public class WebClientTest {
public static void main(String... args) throws InterruptedException {
WebClient webClient = WebClient.builder()
.baseUrl("http://localhost:8080")
.build();
AtomicLong successCount = new AtomicLong();
AtomicLong errorCount = new AtomicLong();
int requestCount = 100000;
System.out.println("test start");
for (int i = 0; i < requestCount; i++) {
if (i % (requestCount / 100) == 0)
System.out.printf("sent: %d requests\n", i);
webClient.get()
.uri("/users/{id}", i)
.retrieve()
.toBodilessEntity()
.subscribe(response -> successCount.incrementAndGet(), throwable -> {
throwable.printStackTrace();
errorCount.incrementAndGet();
});
}
while (true) {
long success = successCount.get();
long error = errorCount.get();
System.out.printf("test results, success: %,d, errors: %,d\n", success, error);
if (success + error == requestCount)
break;
Thread.sleep(1000);
}
}
}
My build.gradle
apply plugin: 'java'
repositories {
mavenLocal()
mavenCentral()
}
dependencies {
compile 'org.springframework:spring-webflux:5.2.5.RELEASE'
compile 'io.projectreactor.netty:reactor-netty:0.9.6.RELEASE'
}
Your Environment
Mac OS Mojave,
- Reactor version(s) used: 0.9.6.RELEASE
- Other relevant libraries versions (eg.
netty, …):4.1.48.Final - JVM version (
java -version):java version "1.8.0_192" - OS and version (eg
uname -a):Darwin ahesofmbr 18.7.0 Darwin Kernel Version 18.7.0: Sun Dec 1 18:59:03 PST 2019; root:xnu-4903.278.19~1/RELEASE_X86_64 x86_64
Issue Analytics
- State:
- Created 3 years ago
- Reactions:5
- Comments:15 (7 by maintainers)
Top Results From Across the Web
Pool#acquire(Duration) has been pending for more than the ...
Session handling error: Pool#acquire(Duration) has been pending for more than the configured timeout of 45000ms · 1. GraphQLClient Microservice ...
Read more >"Pool#acquire(Duration) has been pending for more than the ...
"Pool#acquire(Duration) has been pending for more than the configured timeout of 45000ms" message when creating or deleting Spring Cloud ...
Read more >Configuring timeouts in Spring reactive WebClient
During the test we do use the (Mono) publisher timeout to configure the timeout value as 3 seconds. When we run the test...
Read more >Set a Timeout in Spring 5 Webflux WebClient - Baeldung
The response timeout is the time we wait to receive a response after sending a request. We can use the responseTimeout() method to...
Read more >Web on Reactive Stack - Spring
If you have a Spring MVC application with calls to remote services, try the reactive WebClient . You can return reactive types (Reactor,...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
@echoesbr the fix for this one should be available already with 5.2.6 and above (see the linked issue). The other one in Reactor Netty is in 0.9.5 and above.
@vioao This was closed with a fix in Spring Framework (see the linked issue). If you experience a similar issue please open a new issue with a reproducible example.