Netty connection in pool reset by peer in Spring Cloud Gateway Application

We use Spring Cloud Gateway with Spring Security OAuth2.

SpringBoot version: 2.2.1 Spring Cloud version: Hoxton.Release Netty: reactor-netty-0.9.2.RELEASE

The Spring security OAuth2 filter post request to UAA server, there is load balancer in between. While initially connection all good but after about 10-15 min, Netty used Spring OAuth2 filter got connection reset by peer and the other end is load balancer. If try again, then it will work. But after another 10-15 min, the problem happens again.

The log shows reactor.netty.resources.PooledConnectionProvider uses some active channel but it is actually not active. We are seeking solution to work around it. So far we tried to customize Netty ClientHttpConnector (code as below) to disable pool, but no success. It seems that bean has no effect on Netty used by Spring security.

I uploaded demo project at GitHub link below, https://github.com/spring-cloud/spring-cloud-gateway/issues/1493

That project cannot produce problem at local as there is no load balancer between but it shows the way to customize ClientHttpConnector used by Spring Security to disable Netty Pool does not work.

@bean
public ClientHttpConnector clientHttpConnector() {
return new ReactorClientHttpConnector(HttpClient.create(ConnectionProvider.newConnection()));
}

From Netty side, is there any solution or work around to this issue? Since our project is due soon in production, any work around we are considering. If you could point out where in Netty code, we can disable the pool as a temporary solutions so that we can move forward and revisit this issue later on.

Here is the logs:

2019-12-19 18:07:53.460 DEBUG 7 --- [reactor-http-epoll-1] o.s.w.r.f.client.ExchangeFunctions       : [1b45ced7] HTTP POST https://server-name/oauth/token
2019-12-19 18:07:53.462 DEBUG 7 --- [reactor-http-epoll-4] r.n.resources.PooledConnectionProvider   : [id: 0x2a4d88b0, L:/xxx.xxx.xxx.xxx:38628 - R:xxx.xxx.xxx.xxx:443] Channel acquired, now 1 active connections and 1 inactive connections
2019-12-19 18:07:53.463 DEBUG 7 --- [reactor-http-epoll-4] r.netty.http.client.HttpClientConnect    : [id: 0x2a4d88b0, L:/xxx.xxx.xxx.xxx:38628 - R:xxx.xxx.xxx.xxx:443] Handler is being applied: {uri=https://server-name/oauth/token, method=POST}
2019-12-19 18:07:53.466 DEBUG 7 --- [reactor-http-epoll-4] o.s.http.codec.FormHttpMessageWriter     : [1b45ced7] Writing form fields [grant_type, code, redirect_uri] (content masked)
2019-12-19 18:07:53.471 DEBUG 7 --- [reactor-http-epoll-4] r.n.resources.PooledConnectionProvider   : [id: 0x2a4d88b0, L:/xxx.xxx.xxx.xxx:38628 - R:xxx.xxx.xxx.xxx:443] onStateChange(POST{uri=/oauth/token, connection=PooledConnection{channel=[id: 0x2a4d88b0, L:/xxx.xxx.xxx.xxx:38628 - R:xxx.xxx.xxx.xxx:443]}}, [request_sent])
2019-12-19 18:07:53.474 DEBUG 7 --- [reactor-http-epoll-4] r.netty.http.client.HttpClientConnect    : [id: 0x2a4d88b0, L:/xxx.xxx.xxx.xxx:38628 - R:xxx.xxx.xxx.xxx:443] The connection observed an error, the request will be retried

io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer

2019-12-19 18:07:53.475 DEBUG 7 --- [reactor-http-epoll-4] r.n.resources.PooledConnectionProvider   : [id: 0x9bf92ed9, L:/xxx.xxx.xxx.xxx:38600 - R:xxx.xxx.xxx.xxx:443] Channel acquired, now 2 active connections and 0 inactive connections
2019-12-19 18:07:53.476 DEBUG 7 --- [reactor-http-epoll-4] r.netty.http.client.HttpClientConnect    : [id: 0x9bf92ed9, L:/xxx.xxx.xxx.xxx:38600 - R:xxx.xxx.xxx.xxx:443] Handler is being applied: {uri=https://server-name/oauth/token, method=POST}
2019-12-19 18:07:53.479 DEBUG 7 --- [reactor-http-epoll-4] o.s.http.codec.FormHttpMessageWriter     : [1b45ced7] Writing form fields [grant_type, code, redirect_uri] (content masked)
2019-12-19 18:07:53.492 DEBUG 7 --- [reactor-http-epoll-4] r.n.resources.PooledConnectionProvider   : [id: 0x9bf92ed9, L:/xxx.xxx.xxx.xxx:38600 - R:xxx.xxx.xxx.xxx:443] onStateChange(POST{uri=/oauth/token, connection=PooledConnection{channel=[id: 0x9bf92ed9, L:/xxx.xxx.xxx.xxx:38600 - R:xxx.xxx.xxx.xxx:443]}}, [request_sent])
2019-12-19 18:07:53.507 DEBUG 7 --- [reactor-http-epoll-4] r.n.resources.PooledConnectionProvider   : [id: 0x2a4d88b0, L:/xxx.xxx.xxx.xxx:38628 ! R:xxx.xxx.xxx.xxx:443] Channel closed, now 2 active connections and 0 inactive connections
2019-12-19 18:07:53.517 DEBUG 7 --- [reactor-http-epoll-4] r.n.resources.PooledConnectionProvider   : [id: 0x2a4d88b0, L:/xxx.xxx.xxx.xxx:38628 ! R:xxx.xxx.xxx.xxx:443] onStateChange(POST{uri=/oauth/token, connection=PooledConnection{channel=[id: 0x2a4d88b0, L:/xxx.xxx.xxx.xxx:38628 ! R:xxx.xxx.xxx.xxx:443]}}, [disconnecting])
2019-12-19 18:07:53.518 DEBUG 7 --- [reactor-http-epoll-4] r.n.resources.PooledConnectionProvider   : [id: 0x2a4d88b0, L:/xxx.xxx.xxx.xxx:38628 ! R:xxx.xxx.xxx.xxx:443] Releasing channel
2019-12-19 18:07:53.523 DEBUG 7 --- [reactor-http-epoll-4] r.n.resources.PooledConnectionProvider   : [id: 0x2a4d88b0, L:/xxx.xxx.xxx.xxx:38628 ! R:xxx.xxx.xxx.xxx:443] Channel cleaned, now 1 active connections and 0 inactive connections
2019-12-19 18:07:53.532 DEBUG 7 --- [reactor-http-epoll-4] r.netty.http.client.HttpClientConnect    : [id: 0x9bf92ed9, L:/xxx.xxx.xxx.xxx:38600 - R:xxx.xxx.xxx.xxx:443] The connection observed an error, the request will be retried

io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer

2019-12-19 18:07:53.543 ERROR 7 --- [reactor-http-epoll-4] o.s.w.s.adapter.HttpWebHandlerAdapter    : [177851e0] 500 Server Error for HTTP GET "/login/oauth2/code/myapp?code=Jo87
qf1XIC&state=yMYngnlcVyuy8BgpgFGxZDwo0nmwYg7pa4mcvROTtrU%3D"

io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer
        Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
        |_ checkpoint  Request to POST https://server-name/oauth/token [DefaultWebClient]
        |_ checkpoint  org.springframework.security.oauth2.client.web.server.authentication.OAuth2LoginAuthenticationWebFilter [DefaultWebFilterChain]