Failed to start a new language worker for runtime: dotnet-isolated

See original GitHub issue

We keep getting errors saying Failed to start a new language worker for runtime: dotnet-isolated, it happens about 1-3 times a day (I assume it has to do with scaling and starting up a new worker). I can see in the application insight logs that sometimes it does start accepting requests after it has failed to launch it if I search for the host instance id, but sometimes the host instance id has no more logs after it fails to start the worker.

System.TimeoutException:
   at Microsoft.Azure.WebJobs.Script.Grpc.GrpcWorkerChannel+PendingItem.OnTimeout (Microsoft.Azure.WebJobs.Script.Grpc, Version=4.17.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script.Grpc/Channel/GrpcWorkerChannel.cs:1452)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Script.Grpc.GrpcWorkerChannel+<StartWorkerProcessAsync>d__55.MoveNext (Microsoft.Azure.WebJobs.Script.Grpc, Version=4.17.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script.Grpc/Channel/GrpcWorkerChannel.cs:275)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcFunctionInvocationDispatcher+<InitializeJobhostLanguageWorkerChannelAsync>d__50.MoveNext (Microsoft.Azure.WebJobs.Script, Version=4.17.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script/Workers/Rpc/FunctionRegistration/RpcFunctionInvocationDispatcher.cs:152)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcFunctionInvocationDispatcher+<InitializeJobhostLanguageWorkerChannelAsync>d__49.MoveNext (Microsoft.Azure.WebJobs.Script, Version=4.17.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script/Workers/Rpc/FunctionRegistration/RpcFunctionInvocationDispatcher.cs:144)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcFunctionInvocationDispatcher+<InitializeJobhostLanguageWorkerChannelAsync>d__48.MoveNext (Microsoft.Azure.WebJobs.Script, Version=4.17.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script/Workers/Rpc/FunctionRegistration/RpcFunctionInvocationDispatcher.cs:135)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcFunctionInvocationDispatcher+<>c__DisplayClass55_0+<<StartWorkerProcesses>b__0>d.MoveNext (Microsoft.Azure.WebJobs.Script, Version=4.17.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script/Workers/Rpc/FunctionRegistration/RpcFunctionInvocationDispatcher.cs:215)

Requests still get through since it will start another worker eventually, but it is very annoying to see these logs since we have alarms when we get logs with the ERROR log level. Also I assume this will result in a performance degradation. The response to the caller will be 502 Bad Gateway, which causes issues in integration tests and such.

Checking the logs for the host instance id it seems like the function version is 4.17.3.

I also get this log before the error so it seems like it’s starting something but then times out for some reason:

Host started (3307ms)

Issue Analytics

State:
Created 5 months ago
Reactions:1
Comments:8 (2 by maintainers)

Top GitHub Comments

1reaction

satvucommented, May 8, 2023

We believe this is related to a race condition in worker version 1.13 - this was fixed in https://github.com/Azure/azure-functions-dotnet-worker/pull/1508. Please unpin the host version and upgrade to the latest worker package 1.14.1.

0reactions

gmantricommented, May 19, 2023

I am facing same problems though I am on worker package 1.14.1. I have a .Net 7 isolated function with Service Bus Topic input binding.

        <PackageReference Include="Azure.Identity" Version="1.9.0" />
        <PackageReference Include="Microsoft.Azure.Functions.Worker" Version="1.14.1" />
        <PackageReference Include="Microsoft.Azure.Functions.Worker.ApplicationInsights" Version="1.0.0-preview4" />
        <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.ServiceBus" Version="5.7.0" />
        <PackageReference Include="Microsoft.Azure.Functions.Worker.Sdk" Version="1.10.0" />
        <PackageReference Include="Microsoft.Extensions.Configuration" Version="7.0.0" />
        <PackageReference Include="Microsoft.Extensions.Configuration.AzureAppConfiguration" Version="6.0.1" />
        <PackageReference Include="Microsoft.Extensions.DependencyInjection" Version="7.0.0" />
        <PackageReference Include="Microsoft.Extensions.Hosting" Version="7.0.1" />
        <PackageReference Include="Microsoft.Extensions.Localization" Version="7.0.5" />
        <PackageReference Include="Newtonsoft.Json" Version="13.0.3" />

One of the errors that is getting logged in App Insights is Unhandled exception. System.InvalidOperationException: The gRPC channel URI 'http://:0' could not be parsed..

I even tried pinning the version to 4.16 by going to Function App Configuration > App Configuration and changing the value of FUNCTIONS_EXTENSION_VERSION settings from ~4 to 4.16 but that did not make any difference. On the overview page, I still see the version as 4.21.1.20667.

Can someone tell me what I am doing wrong or how to fix this issue? The code works perfectly fine on my local machine.

Top Results From Across the Web

Failed to start a new language worker for runtime: dotnet ...

I was just dealing with the same problem. I finally fixed it by adding .ConfigureFunctionsWorkerDefaults() to my Program.cs file.

I have an error (Failed to start a new language worker for ...

I have an error (Failed to start a new language worker for runtime: dotnet-isolated) after deploying the isolated Azure Function developed ...

Failed to start a new language worker for runtime: dotnet ...

azure-functions Ensure that the appropriate runtime is specified while creating the Function App. I just ran a NuGet package update for: Microsoft.

Create an Azure Function .NET in the isolated worker model

NET 5 support in Azure Functions, introducing an Isolated Model, running as an out-of-process language worker separate from the Azure Functions runtime.

Azure Functions “Failed to start language worker process for

To fix this, you need to set your function app to use Node 8.11.1. Here is how. Go to your function app, and...