Grpc Error in `dotnet-isolated` in net7 and docker

I have two functions, one is a CronTrigger, which appears to be working fine the other is an OrchestrationTrigger function. This is running in dotnet-isolated, net7, in docker.

I can see in my logs that the cron is launching, it also successfully schedules the other function, i can see the instanceId in the logs.

[2022-12-17T07:00:04.364Z] I "Start: 3fdccdf3-be99-40e3-95c8-53cf6c590910 Microsoft.Azure.Functions.Worker.TimerInfo" {"category":"Recovery.Functions.HelloProviders"}
[2022-12-17T07:00:04.383Z] Start: 3fdccdf3-be99-40e3-95c8-53cf6c590910 Microsoft.Azure.Functions.Worker.TimerInfo
[2022-12-17T07:00:04.616Z] I "Started: 5c832cad72f1403ead887f06e6456f8a..." {"category":"Recovery.Functions.HelloProviders"}
[2022-12-17T07:00:04.617Z] Started: 5c832cad72f1403ead887f06e6456f8a...
[2022-12-17T07:00:04.728Z] Executed 'Functions.CronTest' (Failed, Id=2e972014-43a1-479e-8b10-a412039749a2, Duration=6640ms)
[2022-12-17T07:00:04.728Z] System.Private.CoreLib: Exception while executing function: Functions.CronTest. System.Private.CoreLib: Result: Failure
...

The line where I call WaitForInstanceCompletionAsync(instanceId, ...) is where it throws an exception. Here are the details:

{
    "created": "@1671260404.629909577",
    "description": "Error received from peer ipv4:127.0.0.1:4001",
    "file": "/var/local/git/grpc/src/core/lib/surface/call.cc",
    "file_line": 1070,
    "grpc_message": "Exception was thrown by handler.",
    "grpc_status": 2
}

I’m not sure how this can happen. This is running in a pod, which has both processes…

Example code

using System.Threading;
using Microsoft.Azure.Functions.Worker;
using Microsoft.DurableTask;

namespace Recovery.Functions;

public class HelloExample
{
  private readonly ILogger logger;

  public HelloExample(ILogger<HelloExample> logger) => this.logger = logger;

  [Function(nameof(HelloExample))]
  public async Task<string> Hello([OrchestrationTrigger] TaskOrchestrationContext context, string name)
  {
    var instanceId = context.InstanceId;
    this.logger.LogInformation("Hello Workflow Running: {name} ({instanceId})", name, instanceId);

    return await Task.FromResult($"Hello {name}!");
  }

  [Function(nameof(CronTest))]
  public async Task CronTest(
    [TimerTrigger("* * * * * *")] TimerInfo timerInfo,
    [DurableClient] DurableClientContext durableContext,
    FunctionContext context
  )
  {
    this.logger.Info($"Start: {context.FunctionId} {timerInfo}");
    var instanceId = await durableContext.Client.ScheduleNewOrchestrationInstanceAsync(nameof(HelloExample), "Justin");
    this.logger.LogInformation("Started: {instanceId}...", instanceId);

    var result = await durableContext.Client.WaitForInstanceCompletionAsync(instanceId, CancellationToken.None, true);
    this.logger.LogInformation("Result: {SerializedOutput}", result.SerializedOutput);
    await Task.CompletedTask;
  }
}

Full error and stack

System.Private.CoreLib: Exception while executing function: Functions.CronTest. System.Private.CoreLib: Result: Failure
Exception: System.AggregateException: One or more errors occurred. (Status(StatusCode="Unknown", Detail="Exception was thrown by handler.", DebugException="Grpc.Core.Internal.CoreErrorDetailException: {"created":"@1671260404.629909577","description":"Error received from peer ipv4:127.0.0.1:4001","file":"/var/local/git/grpc/src/core/lib/surface/call.cc","file_line":1070,"grpc_message":"Exception was thrown by handler.","grpc_status":2}"))
---> Grpc.Core.RpcException: Status(StatusCode="Unknown", Detail="Exception was thrown by handler.", DebugException="Grpc.Core.Internal.CoreErrorDetailException: {"created":"@1671260404.629909577","description":"Error received from peer ipv4:127.0.0.1:4001","file":"/var/local/git/grpc/src/core/lib/surface/call.cc","file_line":1070,"grpc_message":"Exception was thrown by handler.","grpc_status":2}")
at Microsoft.DurableTask.Client.Grpc.GrpcDurableTaskClient.WaitForInstanceCompletionAsync(String instanceId, CancellationToken cancellationToken, Boolean getInputsAndOutputs)
at Recovery.Functions.HelloExamples.CronTest(TimerInfo timerInfo, DurableClientContext durableContext, FunctionContext context) in /var/recovery/apps/recovery-provider-api/src/recovery-provider-api/Functions/HelloExamples.cs:line 33
at Microsoft.Azure.Functions.Worker.Invocation.VoidTaskMethodInvoker`2.InvokeAsync(TReflected instance, Object[] arguments) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\VoidTaskMethodInvoker.cs:line 22
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionInvoker`2.<>c.<InvokeAsync>b__6_0(Task`1 t) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionInvoker.cs:line 32
at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionExecutor.ExecuteAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionExecutor.cs:line 44
at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13
at Microsoft.Azure.Functions.Worker.Extensions.DurableTask.DurableTaskFunctionsMiddleware.Invoke(FunctionContext functionContext, FunctionExecutionDelegate next) in /_/src/Worker.Extensions.DurableTask/DurableTaskFunctionsMiddleware.cs:line 18
at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(InvocationRequest request) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Handlers\InvocationHandler.cs:line 82
Stack: at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionInvoker`2.<>c.<InvokeAsync>b__6_0(Task`1 t) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionInvoker.cs:line 32
at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionExecutor.ExecuteAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionExecutor.cs:line 44
at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13
at Microsoft.Azure.Functions.Worker.Extensions.DurableTask.DurableTaskFunctionsMiddleware.Invoke(FunctionContext functionContext, FunctionExecutionDelegate next) in /_/src/Worker.Extensions.DurableTask/DurableTaskFunctionsMiddleware.cs:line 18
at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(InvocationRequest request) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Handlers\InvocationHandler.cs:line 82.