TaskCanceledException exceptions randomly occurring with ResponseBodyCanceled message

Describe the bug

After deploying a service with YARP, we are getting exceptions as seen below. Most requests are successful, but a small portion of them are having this issue. It seems to be affecting the destination controllers that do more processing.

To Reproduce

We’ve been unable to consistently reproduce the issue in our testing environment. It only occurred under a production load. We suspected that it was either timing out or the response was too large, but we were still unable to replicate the issue when testing either of those.

Error message: “ResponseBodyCanceled: Copying the response body was canceled.” Stack trace:

System.Threading.Tasks.TaskCanceledException:
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.ThrowForFailedGetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Net.Http.Http2Connection+Http2Stream.System.Threading.Tasks.Sources.IValueTaskSource.GetResult (System.Net.Http, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Net.Http.Http2Connection+Http2Stream+<ReadDataAsync>d__69.MoveNext (System.Net.Http, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Yarp.ReverseProxy.Forwarder.StreamCopier+<CopyAsync>d__3.MoveNext (Yarp.ReverseProxy, Version=1.1.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)

Further technical details

The service is running on a linux container in an Azure App Service using Yarp.ReverseProxy 1.1.1

The config is below:

"ReverseProxy": {
    "Routes": {
      "clientcontrollers": {
        "ClusterId": "webapi",
        "Match": { "Path": "/Client/{*id}" },
        "Order": -1
      }
    },
    "Clusters": {
      "webapi": {
        "Destinations": {
          "primary": {
            "Address": "https://example.com"
          }
        },
        "HttpRequest": {
          "ActivityTimeout": "00:04:00"
        }
      }
    }
  }

The ActivityTimeout was added after the problem started occurring but hasn’t seemed to change anything