InvalidOperationException: Did not find any initialized language workers

This issue can be seen as a continuation of https://github.com/Azure/azure-functions-host/issues/6931.

Message: Exception while executing function: Functions.keepAlive Did not find any initialized language workers Failed Method: Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcFunctionInvocationDispatcherLoadBalancer.GetLanguageWorkerChannel

The exception is seen in our http triggered, timer triggered and service bus triggered azure functions running on consumption plan written in java running on azure function service plans with Windows as operating system.

Occurrence count

We are currently seeing 600 occurrences of the exception per hour, in a system running a spiked workload of 20 jobs an hour. RetryCount of 6-1 multiplied by servicebus deliveryCount 3 multiplied by the number of azure functions in the job chain 10, allows for a worst case scenario of 3000 occurrences of the exception when running 20 jobs. 

Before the 2nd of July, the occurrence was higher, around 2000 per hour.

Inconsistent bug

The exception does not seem to be consistently occurring on instances where the exception has previously occurred.

First occurrence time

14 of May 2021

Investigative information

Event time | 8/15/2021, 8:31:41 PM (Local time) Message | Exception while executing function: Functions.keepAlive Did not find any initialized language workers Exception type | System.InvalidOperationException Failed method | Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcFunctionInvocationDispatcherLoadBalancer.GetLanguageWorkerChannel HostInstanceId | d6a3afbd-6481-4124-ba5f-f2150ea86faa InvocationId | 889c1000-980c-4676-b32d-ac3b72885a58 prop__functionName | Functions.keepAlive

Repro steps

A. Deploy an http or service bus or Timer triggered azure function in a function app running in consumption plan with the configurations specified in the following section. B. Load test the azure function to cause a scale out. C. Language worker initialisation exception is thrown.

System Settings

Function App Service Plan

Operating System: Windows Location: West Europe
 Zone redundant: Disabled Status: Ready

Function App Details - Site Config

“linuxFxVersion”: “”, “javaVersion”: “1.8”, “location”: “West Europe”, 
"minTlsVersion": “1.2”, 
"netFrameworkVersion": “v4.0”, 
"maxNumberOfWorkers": null, 
"numberOfWorkers": 1, “phpVersion”: “5.6”, “powerShellVersion": “”, “preWarmedInstanceCount": 0, “pythonVersion”: “”, (Revealed by running az functionapp show -g prodb-publicapi -n prodb-publicapi)

Attempted fixes

Specifying the java-version to 1.8, was previously unspecified (As specified on the 6931 issue). Migrate to a new scale unit on the Function App (done by an azure supporter currently only on a non failing function app).

Related issues

This issue can be seen as a continuation of https://github.com/Azure/azure-functions-host/issues/6931. https://github.com/Azure/azure-functions-host/issues/6936 https://github.com/Azure/azure-functions-host/issues/5367 https://github.com/Azure/azure-functions-host/issues/5233

Microsoft Employees that has already verified and worked with the issue, the recent supporter at the top of the list

Ritika Singhal Luis Ramirez Harshita Bajpai @FinVamp1 Finbar Ryan

The following conclusion was written by an Azure App Service Support Engineer on the 28th of June

Summary In the occasions where the Failed to Initialize Language Worker is being logged what we see is: The Azure Functions Application is started on a new worker The Functions Host starts It sends a request out to start the Java Language worker process. This process starts but never replies back to the Functions host and so the Language Worker process start request times out. This process continues to run and then shuts down about 30 minutes later.

Next Steps The start of the Language worker process should initialize the Functions Java Language Worker process and then reply back to the Functions Host. As this is not happening for a lot of customers we would like your help in understanding the specifics about your application.