Worker crash during `FunctionEnvironmentReload` (specialization) blocks for the full 30s timeout instead of failing fast

[brettsam]

Summary

When a language worker process crashes while handling a FunctionEnvironmentReloadRequest (i.e. during specialization), the host does not detect the crash for the purpose of the reload. Instead, SpecializeAsync waits the full EnvironmentReloadTimeout (default 30s) before failing with a TimeoutException, even though the worker process died within milliseconds. This adds a ~30s stall to specialization/cold start whenever the worker crashes on reload.

Environment

• Component: Microsoft.Azure.WebJobs.Script.Grpc (WebHost worker channel / specialization)
• Applies to placeholder → specialize (env reload) path on Linux/Windows.

Expected behavior

If the worker process exits while a FunctionEnvironmentReload is pending, specialization should fail fast (surface the worker exit as the failure cause) rather than waiting for the reload timeout to elapse.

Actual behavior

Specialization blocks for the full EnvironmentReloadTimeout (30s) and then fails with a generic TimeoutException, masking the real cause (worker crash) and delaying recovery/host restart.

Root cause

SpecializeAsync awaits the reload completion source, which is only completed on the gRPC response or on the reload timeout — a worker process crash never completes it.

Trace (paths from dev):

1. WebHostRpcWorkerChannelManager.SpecializeAsync() awaits rpcWorkerChannel.SendFunctionEnvironmentReloadRequest().
   • WebHostRpcWorkerChannelManager.cs → envReloadRequestResultSuccessful = await rpcWorkerChannel.SendFunctionEnvironmentReloadRequest();
2. WorkerChannel.SendFunctionEnvironmentReloadRequest() returns _reloadTask.Task.
   • WorkerChannel.cs:710 → return _reloadTask.Task;
3. _reloadTask (a TaskCompletionSource<bool>, WorkerChannel.cs:85) is completed in exactly two places:
   • Success: on receiving FunctionEnvironmentReloadResponse → SetResult (WorkerChannel.cs:499/503).
   • Timeout: the PendingItem registered for the response (WorkerChannel.cs:698-699) uses CancellationTokenSource.CancelAfter(EnvironmentReloadTimeout). On expiry, PendingItem.OnTimeout throws TimeoutException → fault handler HandleWorkerEnvReloadError → _reloadTask.SetException (WorkerChannel.cs:1367).
4. A worker crash flows through a completely separate path that never touches _reloadTask:
   • WorkerProcess.OnProcessExited (WorkerProcess.cs:219) → RpcWorkerProcess.HandleWorkerProcessExitError → PublishNoThrow(new WorkerErrorEvent(...)) (RpcWorkerProcess.cs:84-86).
   • WorkerErrorEvent is consumed by RpcFunctionInvocationDispatcher.WorkerError (RpcFunctionInvocationDispatcher.cs:476) for dispatcher-level restart handling.
   • Nothing in this path completes/faults _reloadTask, so the pending reload is orphaned until the 30s timer fires.

Proposed fix

When the worker process exits while a reload is still pending, fault the reload completion source so specialization fails fast:

• Have the WorkerChannel observe its own process exit / WorkerErrorEvent, and if _reloadTask is still pending, call _reloadTask.TrySetException(...) (e.g. wrapping the WorkerProcessExitException).
• Use TrySetException/TrySetResult (not SetException/SetResult) throughout the reload-completion paths to avoid races with the timeout/response callbacks completing the TCS first.

Impact

• Adds ~30s to specialization/cold start on every worker crash-during-reload.
• Masks the real failure cause (TimeoutException instead of the worker exit), making diagnosis harder.

Notes

• EnvironmentReloadTimeout default is 30s (WorkerProcessCountOptions.cs:48).
• Related deterministic-timeout behavior: Function app with 'Warmup' function  #10169. (This issue is the "worker actually crashed" variant of that timeout window.)