WebClient fails with Pool#acquire(Duration) has been pending for more than the configured timeout of 45000ms after 33000 request in loop

See original GitHub issue

I’ve tried write a simple load test with WebClient. One thread with simple loop, 100k iterations, but after 2000 or 3000 cycles I receive a message:

Apr 09, 2020 6:00:38 AM io.netty.util.ResourceLeakDetector reportTracedLeak
SEVERE: LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
Created at:
    io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:363)
    io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
    io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
    io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:139)
    io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114)
    io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:147)
    io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
    io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
    io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
    io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.lang.Thread.run(Thread.java:748)

After all 100k iterations, I’ve received only 33k successful responses, and no one was failed. My code written to wait for all responses and prints success/failed requests after one second of sleep. After 45 seconds, I’ve received other 67k failed requests with exception:

reactor.netty.internal.shaded.reactor.pool.PoolAcquireTimeoutException: Pool#acquire(Duration) has been pending for more than the configured timeout of 45000ms
    at reactor.netty.internal.shaded.reactor.pool.AbstractPool$Borrower.run(AbstractPool.java:334)
    Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):
    |_ checkpoint ⇢ Request to GET http://localhost:8080/users/50725/5362 [DefaultWebClient]
Stack trace:
        at reactor.netty.internal.shaded.reactor.pool.AbstractPool$Borrower.run(AbstractPool.java:334)
        at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
        at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

And my stats looks like

test results, success: 33,085, errors: 66,914 

Note that 33085 + 66914 = 99999

Steps to Reproduce

I wrote a simple app with spring-webflux:5.2.5 and reactor-netty:0.9.6:

public class WebClientTest {

    public static void main(String... args) throws InterruptedException {
        WebClient webClient = WebClient.builder()
                .baseUrl("http://localhost:8080")
                .build();

        AtomicLong successCount = new AtomicLong();
        AtomicLong errorCount = new AtomicLong();
        int requestCount = 100000;
        System.out.println("test start");
        for (int i = 0; i < requestCount; i++) {
            if (i % (requestCount / 100) == 0)
                System.out.printf("sent: %d requests\n", i);

            webClient.get()
                    .uri("/users/{id}", i)
                    .retrieve()
                    .toBodilessEntity()
                    .subscribe(response -> successCount.incrementAndGet(), throwable -> {
                        throwable.printStackTrace();
                        errorCount.incrementAndGet();
                    });
        }


        while (true) {
            long success = successCount.get();
            long error = errorCount.get();
            System.out.printf("test results, success: %,d, errors: %,d\n", success, error);
            if (success + error == requestCount)
                break;

            Thread.sleep(1000);
        }
    }
}

My build.gradle

apply plugin: 'java'

repositories {
    mavenLocal()
    mavenCentral()
}

dependencies {
    compile 'org.springframework:spring-webflux:5.2.5.RELEASE'
    compile 'io.projectreactor.netty:reactor-netty:0.9.6.RELEASE'
}

Your Environment

Mac OS Mojave,

  • Reactor version(s) used: 0.9.6.RELEASE
  • Other relevant libraries versions (eg. netty, …): 4.1.48.Final
  • JVM version (java -version): java version "1.8.0_192"
  • OS and version (eg uname -a): Darwin ahesofmbr 18.7.0 Darwin Kernel Version 18.7.0: Sun Dec 1 18:59:03 PST 2019; root:xnu-4903.278.19~1/RELEASE_X86_64 x86_64

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Reactions:5
  • Comments:15 (7 by maintainers)

github_iconTop GitHub Comments

1reaction
violetaggcommented, Jul 10, 2020

@echoesbr the fix for this one should be available already with 5.2.6 and above (see the linked issue). The other one in Reactor Netty is in 0.9.5 and above.

1reaction
violetaggcommented, Jul 8, 2020

@vioao This was closed with a fix in Spring Framework (see the linked issue). If you experience a similar issue please open a new issue with a reproducible example.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Pool#acquire(Duration) has been pending for more than the ...
Session handling error: Pool#acquire(Duration) has been pending for more than the configured timeout of 45000ms · 1. GraphQLClient Microservice ...
Read more >
"Pool#acquire(Duration) has been pending for more than the ...
"Pool#acquire(Duration) has been pending for more than the configured timeout of 45000ms" message when creating or deleting Spring Cloud ...
Read more >
Configuring timeouts in Spring reactive WebClient
During the test we do use the (Mono) publisher timeout to configure the timeout value as 3 seconds. When we run the test...
Read more >
Set a Timeout in Spring 5 Webflux WebClient - Baeldung
The response timeout is the time we wait to receive a response after sending a request. We can use the responseTimeout() method to...
Read more >
Web on Reactive Stack - Spring
If you have a Spring MVC application with calls to remote services, try the reactive WebClient . You can return reactive types (Reactor,...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found